Table of contents 1.Introduction 1.1 Purpose and scope of the meta-evaluation 2.What was the expected outcome of the intervention? 2.1 Description of the intervention and its objectives 2.2 Point(s) of comparison 3.How has the situation evolved over the evaluation period? 4.Evaluation findings (analytical part) 4.1 To what extent was the intervention successful and why? 4.2 How did the EU intervention make a difference? 4.3 Is the intervention still relevant? 4.4 Is the current framework for evaluating interventions appropriate? 5.What are the conclusions and lessons learned? 5.1 Conclusions 5.2 Lessons learned Annex I: Procedural Information Annex II. Methodology and Analytical models used Annex III. Evaluation matrix and, where relevant, Details on answers to the evaluation questions (by criterion)

Annex IV. Overview of benefits and costs and, where relevant, table on simplification and burden reduction. Annex V. Stakeholders consultation - Synopsis report Bibliography



Glossary Term, abbreviation or acronym Meaning or definition BoPF Balance of Payments Facility: the EU’s facility for providing medium-term financial assistance for Member States’ balances of payments pursuant to Council Regulation (EC) No 332/2002. BR(G) Better Regulation (Guidelines): the Commission’s policy framework for ensuring evidence-based and transparent law-making based on the views of those who may be affected by the laws in question. Comitology Arrangements pursuant to Regulation (EU) No 182/2011 for Commission implementing acts to be submitted to a committee of Member State representatives either pronouncing itself or voting on the draft before it enters into force. CPI Consumer Price Index. DG ECFIN Directorate-General for Economic and Financial Affairs of the European Commission. ECJ European Court of Justice. EEAS European External Action Service. EFSM European Financial Stabilisation Mechanism: the EU’s facility for providing financial assistance to Member States pursuant to Council Regulation (EU) No 407/2010. EIB European Investment Bank. ENI European Neighbourhood Instrument. ENPI European Neighbourhood Policy Instrument. Financial programming Determination of a target for the fiscal and monetary stance to produce a certain economic outcome (e.g. the observance of external financing constraints given a certain exchange rate level) on the basis of a quantitative model of key inter-relationships in an economy. Framework regulation A regulation containing the general legal provisions governing the matter concerned, e. g. procedures, scope, purpose. GDP Gross domestic product. Global Financial Safety Net The global network of crisis-financing instruments encompassing self-insurance (reserves); bilateral arrangements (swap lines between central banks during periods of stress); regional arrangements whereby members stand ready to support one another; and a multilateral system with the IMF at its centre. IFI International financial institution. IMF International Monetary Fund. IPA Instrument for Pre-Accession Assistance. Joint Declaration Joint Declaration by the European Parliament and the Council adopted together with the decision providing further macro-financial assistance to Georgia (Decision No 778/2013/EU (OJ L 218, 14.8.2013, p. 15). LFA Loan Facility Agreement. MFA Macro-financial assistance. MFF Multiannual Financial Framework: the seven-year framework that sets out the annual expenditure ceilings for various policy areas. MoU Memorandum of understanding. NDICI (Regulation) Regulation (EU) 2021/947 of the European Parliament and of the Council of 9 June 2021 establishing the Neighbourhood, Development and International Cooperation Instrument – Global Europe (OJ L 209, 14.6.2021, p. 1). OLP Ordinary legislative procedure pursuant to Article 294 TFEU. OPC Open public consultation. Public financial management The set of all components of a country’s budget process, including strategic planning, a medium-term expenditure framework, annual budgeting (formulation and execution), as well as revenue management, procurement, control, accounting, reporting, monitoring and evaluation, audit and oversight. SDR Special drawing right: IMF-created international reserve asset exchangeable for the freely usable currencies of IMF members. TEU Treaty on the European Union. TFEU Treaty on the Functioning of the European Union. 1.Introduction 1.1 Purpose and scope of the meta-evaluation Macro-financial assistance (MFA) is a form of EU financial aid for partner countries experiencing (or threatened by) a balance of payments crisis, helping to restore their external stability and to return their economies to a sustainable path. Taking the form of medium- or long-term loans or grants (or an appropriate combination of the two), and normally complements financing provided by the international community in the context of a financial arrangement with the International Monetary Fund (IMF). Support can be provided only if beneficiaries respect effective democratic mechanisms. Disbursements are conditional on the implementation of policy reforms agreed between the Commission on the EU’s behalf and the beneficiary country and laid down in a memorandum of understanding (MoU) (European Commission, 2021). Since its inception 30 years ago, MFA has been increasingly used by the EU in its dealings with countries outside the EU and is an important part of its contribution to the Global Financial Safety Net under G20 rules. Initially MFA was extended mostly to countries of central and eastern Europe (some of which have since joined the EU), but it has progressively been extended to other countries in the former Soviet Union and the Mediterranean. MFA volumes increased substantially following the global economic and financial crisis of 2008-2009, which profoundly affected the emerging economies of EU neighbourhood countries. Throughout 2014-2020, MFA has been instrumental in supporting partner countries in the eastern and southern neighbourhood facing conflict and challenging political transitions. In 2020-2022, EUR 3 billion in MFA loans was made available to help 10 partner countries limit the economic fallout of the COVID-19 pandemic. More recently, in 2022, with the escalation of the conflict and the Russian war of aggression against Ukraine, Ukraine requested new MFA support. It received EUR 1.2 billion in emergency MFA (disbursed in March 2022) and, at the time of writing, EUR 6 billion in exceptional MFA (EUR 1 billion disbursed in August and EUR 5 billion disbursed from October to December 2022). The geopolitical, security-related and economic challenges beyond the EU’s borders require a strong and determined action. To rise to new (and old) challenges, the current Commission has identified the need to strengthen the role of Europe in the world (Von der Leyen, 2019). To do this, the EU must act quickly and efficiently to support key partner countries and territories facing external financing difficulties. MFA is one way of doing this. The COVID-19 pandemic illustrates this perfectly. The pandemic challenges partner countries and their financial capacity like never before. Against this backdrop, the EU granted a total of EUR 3 billion in MFA to 10 partner countries in spring 2020 (Decision (EU) 2020/701). This decision and the speed it was implemented with showed that MFA continues to be used extensively by the EU in its dealings with countries outside the EU, in line with its geopolitical ambitions. MFA has also demonstrated its worth as a flexible crisis instrument enabling the EU to swiftly and substantially support Ukraine following Russia’s unprovoked and unjustified invasion of the country at the beginning of 2022. Already since 2014, Ukraine has received an unprecedented amount of MFA. In 2014-2019, a total of EUR 4.4 billion was made available to Ukraine in four MFA operations. This is the most EU MFA ever granted to a non-EU country in such a short period of time. Following Russia’s unprovoked and unjustified invasion of Ukraine, an additional EUR 7.2 billion of MFA loans were made available at record speed. This is testament to the EU’s unwavering support for Ukraine. Unlike other EU external action instruments, MFA operations are adopted and implemented directly on the basis of the treaties, without a specific legal framework. Since the Treaty on the European Union (TEU) and the Treaty on the Functioning of the European Union (TFEU) entered into force on 1 December 2009, Article 212 TFEU (and for developing countries Article 209 TFEU) are the legal basis for MFA and govern it.. The European Parliament and the Council authorise each MFA operation on the basis of a dedicated Commission proposal, in accordance with the ordinary legislative procedure (OLP, as set out in Article 294 TFEU). In urgent cases, MFA operations can be authorised by the Council alone, in a non-legislative procedure, always on the basis of a Commission proposal, pursuant to Article 213 TFEU. For a crisis instrument, the decision-making process for MFA is considered rather lengthy. The first meta-evaluation of past MFA operations, undertaken in 2009, identified delays in adopting MFA decisions as a major shortcoming of the instrument. To remedy this, the Commission put forward a proposal for a Framework Regulation for MFA in July 2011, aimed at expediting decision-making and aligning the management of the MFA instrument with that of other external instruments (European Commission, 2011). However, the proposed approach was not acceptable to the co-legislators, which modified the proposal to the extent that it lost its “raison d’être”, and the Commission ultimately withdrew the proposal; lawfully, as the Court of Justice later held (ECJ, 2015). As a result, there is no framework regulation for MFA policy and legislative decisions on individual MFA operations continue to be adopted by the EP and the Council on a case-by-case basis in the OLP. Recent experience with the COVID-19 MFA package and Ukraine have shown that the current MFA set-up lends itself to swift adoption, if all institutions agree to make full use of the available procedural flexibilities in particularly urgent cases. In line with the applicable legal and political framework , ex-post evaluations of each individual MFA operation are undertaken. This means that a steady stream of evaluations, not necessarily in sync with the MFF cycle, is a distinct feature of MFA. By 2020, 15 such evaluations had been completed since the first meta-evaluation of MFA operations in 2009. In 2020/21, the Commission’s Directorate-General for Economic and Financial Affairs (DG ECFIN) carried out a new meta-evaluation covering 15 MFA evaluations completed in the last 10 years. The exercise, coinciding with the MFA instrument’s thirtieth anniversary, took stock of previous evaluation results and aims to support organisational learning both in terms of the design and implementation of the MFA instrument and the methodology used to evaluate individual operations. The meta-evaluation will also support transparency and accountability and help lay the groundwork for any possible improvements in the instrument’s design and/or implementation that might be necessary in order to even better support the need for a stronger ‘Global Europe’ in line with President von der Leyen’s vision. An external evaluation study has been undertaken to help inform the meta-evaluation (DG ECFIN, 2020). The study began in December 2020 and ended in October 2021. Its outcome is supplemented by input from a high-level seminar on the experience of 30 years of MFA that took place in November 2021 to flesh out the evidence base for the meta-evaluation. Against this backdrop, and on the basis of the 15 evaluations mentioned above, the meta-evaluation: 1.assesses the principles and characteristics of the MFA instrument in line with the Better Regulation criteria of relevance, effectiveness, efficiency, coherence, value added, and taking into account the instrument’s ability to respond to EU external action priorities, as well as the day-to-day management of MFA interventions;

2.assesses the overarching methodological framework applied across the evaluations and identifies areas where there is room for improvement. The meta-evaluation’s methodological approach is threefold: 1.participatory and inclusive data collection and analysis in a (key) stakeholder consultation programme (semi-structured interviews and a public consultation), a validation workshop preceded by a roundtable with the Commission staff in charge of MFA, and a high-level seminar; 2.a mix of qualitative and quantitative input, with qualitative input consisting mainly of document analysis (chiefly the evaluation reports of the 15 operations mentioned above) and stakeholder input, and quantitative data supporting the qualitative information as much as possible; 3.triangulation: the information and data collected from a range of different sources collectively provide answers to DG ECFIN’s evaluation questions. A number of limitations were experienced during the completion of the study and consequently the preparation of the meta-evaluation, including data availability and access to stakeholders (Valdani, Vacari e Associati & Centre for Strategy and Evaluation Services, Oxford Analytics, 2021). Such limitations have undoubtedly had a bearing on the quality of the exercise, but do not affect the robustness or reliability of the meta-evaluation’s conclusions, which serve as a useful basis for reflections on the instrument. The limitations and mitigating factors are elaborated on in Annex II. 2.What was the expected outcome of the intervention? 2.1Description of the intervention and its objectives Since 1990, the EU (and its predecessors) has been providing MFA to partner countries, primarily in its neighbourhood. Although the EU has been using the instrument relatively regularly, it continues to be meant to be an ad hoc financing tool in the EU’s external policy toolkit, triggered by sudden economic and financial developments in recipient countries. This means that instead of having a specific legal basis, operations have been based on the EU’s ‘flexibility clause’, requiring a unanimous decision in the Council without a formal say for the European Parliament (European Commission, 2021). In order to nonetheless provide a framework for those ad hoc decisions, the Council formulated a set of criteria to determine the basic features of MFA operations at an informal meeting in Genval, Belgium (the Genval criteria) in 1993. It revised these criteria in 1996 and 2002 (European Commission, 2011). By a Joint Declaration in 2013, the European Parliament and the Council has provided guidance on the MFA instrument. While not legally binding, it takes stock of the main features of MFA. It was drafted in the aftermath of the entry into force of the Lisbon Treaty in 2009, which gave the European Parliament full co-legislator powers on MFA. The Declaration, based on the Genval criteria, albeit clarified and adapted reflects the co-legislators’ shared understanding. In line with these criteria, MFA is an exceptional financing instrument of untied and undesignated balance-of-payments support for eligible third countries and territories. Its objectives are: 1.to restore a sustainable external finance situation for eligible countries and territories facing external financing difficulties; 2.to underpin the implementation of a policy programme that contains far-reaching adjustment and structural reform measures designed to improve countries’ and territories’ balance of payment positions, in particular over the programme period; 3.to strengthen the implementation of relevant EU programmes and agreements with the EU. Box: The principles of the Joint Declaration The Joint Declaration of the European Parliament and the Council sets out the principles to be followed for co-legislators when agreeing on MFA. These principles are based on the earlier Genval criteria and incorporate some parts of the 2011 Commission proposal (COM(2011) 396 final) for a framework regulation. (1)Aim of assistance: This principle sets out the underlying context for MFA. The assistance should be exceptional, with the aim of providing balance-of-payments support to help establish a sustainable situation for countries and territories facing difficulties in this area. It should be focused on the financing gap a country faces, taking into account the assistance received from other international partners, in particular the IMF. MFA operations should be short-term. (2)Geographic scope: This principle indicates which countries or territories are eligible to apply for financial assistance, namely candidate and potential candidate countries for membership of the EU, or countries in the scope of its neighbourhood policy. Other countries might be considered eligible in exceptional cases. (3)Form of assistance: Assistance should mostly be in the form of loans, but when appropriate can be provided in the form of grants or a combination of the two. (4)Financial provisions: This principle establishes the financial provisions and how they have to be in line with the EU Multiannual Financial Framework.

(5)Amount of the assistance: The amount of financial assistance should be based on the country’s/territory’s financing needs or gaps, consistent with a sufficient and defendable level of foreign exchange reserves. (6)Conditionality: MFA is conditional on the (prospective) recipient’s: a.respecting effective democratic mechanisms, including a multiparty parliamentary system and the rule of law, and on its guaranteeing respect for human rights (political precondition); b.having in place a non-precautionary financial arrangement with the IMF and (in the disbursement phase) a satisfactory track record of implementing the relevant IMF-supported economic adjustment programme; c.implementing specific structural reforms (policy conditions) that the recipient would have to undertake in a set timeframe, agreed between the Commission and the recipient in an MoU. (7)Procedure: The EU decision-making procedure sets out the details of the different roles and steps in providing financial assistance. These consist of the ordinary legislative procedure (OLP) for the adoption of a decision on a proposal by the Commission, and comitology for the adoption of the MoU (without veto powers for Member States in the case of operations below EUR 90 million and with veto powers for operations above that), as well as the Commission Decision with respect to the grant and/or loan agreement. Principles (8)-(11) focus on operational details with regard to implementation and financial management, disbursements, support measures and the protection of the EU’s financial interests. Principles (12) and (13) make provision for regular reporting on and evaluation of MFA operations. The combination of the objectives mentioned above makes MFA a singular instrument. ·On the one hand, it is a traditional instrument for addressing temporary external financing difficulties and crises. In that sense, it mirrors and is closely linked to IMF interventions with the same purpose, albeit intended as a complementary measure to create financial stability. ·On the other hand, unlike the IMF, the EU is an explicitly political actor, and in line with Article 205 TFEU and Article 21 TEU, MFA is ultimately a political action. As such, it is part and parcel of a framework of tools, instruments and policies designed to also achieve the EU’s broader objectives and promote its values. It therefore follows that MFA combines parts of a balance-of-payments financing tool (such as the IMF’s) with traditional instruments of external financial cooperation, whilst being distinct from them both. In contrast to IMF support, MFA explicitly requires the fulfilment of political preconditions. In contrast to other EU external financial cooperation instruments, MFA operations continue to be authorised by means of the OLP, thereby remaining outside of the scope of the Commission’s implementing powers under the EU’s Neighbourhood, Development and International Cooperation Instrument – Global Europe (NDICI). MFA typically comes into play when an unsustainable mix of economic policies or an economic shock have given rise to external financing imbalances that result (or risk resulting) in an undesired currency depreciation. MFA operations therefore follow a traditional pattern of seeking to correct such an unsustainable policy mix (or other causes of a shock) whilst easing economic adjustment by means of financial assistance. It targets the policy areas that are relevant to macroeconomic stability in general and aims to improve them, in line with the following diagram. Figure 1: MFA intervention logic ·The Commission (on behalf of the EU) provides financial assistance, having agreed with the beneficiary on a set of (structural) policy conditions underpinning the macroeconomic adjustment taking place. Technical assistance may also be agreed (Inputs). ·The beneficiary begins the process of implementing the reforms agreed on. In the 15 operations evaluated, reforms tended to focus on public sector governance, the fight against corruption and money laundering, the energy sector, the business climate, the financial sector, and the social safety net and labour market. ·During and after an MFA operation, the policy reforms and financial assistance should help stabilise the beneficiary’s macroeconomic situation, including with regard to their balance-of-payment position and (as necessary) public finances. Ideally, these effects should last for a long period, underpinned by structural improvements in the efficiency and effectiveness of the development and implementation of economically relevant policies. However, the specific intended effects depend on the operation and the country/economy concerned. For reasons already mentioned, the intervention logic and the operation’s operational objectives need to be placed in the context of the MFA’s ends and purposes as a way for the EU to carry out external action. These include, among others, safeguarding EU values and fundamental interests, consolidating and supporting democracy and the rule of law, encouraging the integration of all countries into the world economy, and promoting an international system based on closer multilateral cooperation and good global governance (Article 21 TEU). These objectives (except in individual cases) are not part of the operational objectives of MFA operations and therefore of the intervention logic. Rather, they are preconditions for a country’s benefiting from MFA support.

It must also be stressed that MFA is usually part of a stabilisation operation incorporating, alongside the EU, at least the IMF and most often also the World Bank Group and in some cases other bilateral and multilateral international partners. In the case of the IMF, the link is particularly close given the need for parallel IMF support (Principle 6 of the Joint Declaration) and the requirement for the amount of assistance to be based on a country’s needs (Principle 5). At the same time, given the IMF’s own ceilings (as a multiple of a country’s share in its capital, the quota (IMF, 1944)), its interventions can equally be contingent on an MFA operation as it is necessary to ensure that the stabilisation operation is fully financed. This link also results in a certain division of labour: Commission services rely on the IMF its core areas of expertise, most notably in agreeing with the beneficiary country the macroeconomic stance required to underpin the adjustment to external financing constraints (the financial programming exercise). The IMF’s own policies require it to comprehensively cover all reform requirements (including structural ones) necessary for the standalone success of its intervention (IMF, 2005). Since MFA operations are conditional on a continuously satisfactory track record in implementing IMF-supported policy programmes, IMF conditions (unless waived) are in practice incorporated into the MFA requirements. For its part, the Commission provides the IMF with its country-specific expertise on structural reform priorities, as well as a set of draft reforms designed to support the implementation of the IMF’s conditions. Similarly, MFA usually comes into play alongside other EU instruments. Following on from the European Neighbourhood Policy Instrument (ENPI) and the European Neighbourhood Instrument (ENI – 2014-2020), the Neighbourhood, Development and International Cooperation Instrument (NDICI) is to give neighbourhood partner countries support during the 2021-2027 period; in accession countries, the relevant instrument is the Instrument for Pre-Accession Assistance (IPA). The EU is also active regionally through EIB lending. 2.2Point(s) of comparison As Section 1 explains, MFA operations are systematically evaluated after they end, notably to assess (i) the impact of MFA on beneficiary countries’ economies and in particular on the sustainability of their external position and (ii) the operations’ added value. These evaluations are also supposed to provide the following insights into the EU’s financial assistance: ·to what extent the ex ante considerations determining the design and terms of each operation have been appropriate, taking due account of the economic, political and institutional context; ·to what extent the outcome of the operations has achieved the objectives set in the legislative decisions. The ex post evaluations consist of an ex post assessment of the MFA operations in question, focusing on the macroeconomic and structural adjustment measures that the Commission, on the EU’s behalf, agreed with each beneficiary country in an MoU. The MoU’s policy conditions are based on the economic stabilisation and reform programme endorsed by the authorities of the beneficiary country and are consistent with its agreements with the IMF. In line with the division of labour mentioned above that has evolved over time between the Commission and the IMF, the MoU does not include specific macroeconomic indicators and targets. This is because the Commission relies for these on the IMF’s macro-framework agreed with the country receiving MFA. Instead, the MoUs specify that disbursement of MFA is conditional on a satisfactory track record of implementation of the policy programme agreed between the country and the IMF. Given this is the case, a precise assessment of its standalone impact is difficult to make. As explained above, MFA is an external financial cooperation instrument (serving both economic and external policy objectives), implemented alongside an IMF intervention and the EU’s regular financial operations, typically as part of a stabilisation operation involving other international institutions. This means that the operations’ effects on macroeconomic variables over time cannot be attributed solely to MFA. In addition, as MFA is a short-term crisis-related instrument spanning a maximum of 2 to 3 years, it is not possible to quantify its impact in terms of indicators/milestones beyond the timeframe of the MFA operations themselves. This makes it difficult to capture the impact of some of the medium-term reforms. Despite these constraints, past evaluations have traditionally looked at a certain number of macroeconomic indicators. These included: ·indicators for gauging external stability: developments of international reserves and current account balances; ·indicators for gauging internal stability: inflation levels, GDP growth, government structural balances and debt levels. Besides the constraints outlined above, some of these indicators (such as GDP growth, government debt levels, current account balances) also raise conceptual questions. Assessing significant impacts on GDP growth and government debt is therefore ambitious even for a combined MFA/IMF operation, while government structural balances are beyond the scope of MFA. Finally, current account balances also behave peculiarly, tending to exhibit some automatic self-correction over the cycle. This makes it difficult to assess and isolate the impact of MFA operations on such indicators as their evolution depends on multiple factors. Further analysis may therefore be warranted to fine-tune the Guidelines for the Ex Post Evaluation of Macro-financial Assistance Operations that will be included in the new ECFIN framework contracts for the provision of evaluation and evaluation-related services. 3.How has the situation evolved over the evaluation period? Since the 2009 meta-evaluation, the main features of the MFA instrument have remained unchanged, with some limited changes introduced following the adoption of the Lisbon Treaty. This section provides an overview of the operations evaluated since 2010, the applicable decision-making procedures, and the amounts and financial terms of the assistance. It also describes the budgetary arrangements governing the instrument as well as the conditions and compliance with them. MFA operations 2010-2020 MFA amounts made available to non-EU countries increased from EUR 1.2 billion in 2000-2008 to EUR 6.3 billion in 2009-2019. Political and economic upheaval in Arab Mediterranean partner countries has led to greater demand for MFA since 2012, with operations implemented in Jordan and Tunisia. From 2014, the conflict in eastern Ukraine and the marked deterioration of the country’s economic and financial situation triggered an unprecedented level of MFA support. In 2014-2019, a total of EUR 4.4 billion was made available to Ukraine under four MFA operations, the largest amount of financial assistance ever granted to a non-EU country in such a short period of time. All but two of these operations have already been evaluated and are included in the meta-evaluation. During the 2010-2020 period, the bulk of MFA funds (71%) was used to support eastern neighbourhood countries, followed by southern neighbourhood countries (24%), (pre)-accession countries and other countries (5%). The bulk of MFA funds was provided in loans (95%), the rest in grants (5%).

Of these 17 MFA operations, in total 15 have been completed and evaluated since the last meta-evaluation in 2009/10. Table 1: MFA operations evaluated over the 2010-2020 period Operation Timeframe↓ Year of evaluation Georgia 06 2006-08 2010 Kosovo 06 2006-10 2012 Moldova 07 2007-08 2010 Lebanon 07 2007-10 2012 Georgia 09 2009-10 2012 Serbia 09 2009-11 2013 Bosnia and Herzegovina 09 2009-13 2015 Moldova 10 2010-12 2013 Armenia 09 2011-12 2013 Kyrgyz Republic 13 2013-16 2019 Georgia 13 2013-17 2019 Jordan 13 2014-15 2017 Ukraine 10 and 14 2014-15 2017 Tunisia 14 2014-17 2019 Ukraine 15 2015-18 2020 All 2006-18 2010-20 Many operations were also authorised in the wake of the COVID-19 pandemic, and more recently in the wake of Russia’s unprovoked and unjustified invasion of Ukraine. In response to the pandemic, the Commission adopted a proposal for a EUR 3 billion MFA package for 10 enlargement and neighbourhood partner countries in spring 2020. This was endorsed by the co-legislators in May 2020. At the time of its adoption, it represented the largest volume of MFA the EU ever made available to partner countries at a single time. In response to the 2022 invasion of Ukraine, an additional EUR 7.2 billion of MFA loans was made available to Ukraine at record speed. The COVID-19 package of operations was only fully completed by mid-2022 and support for Ukraine is ongoing. Both sets of operations therefore remain to be evaluated. They provide valuable insights into, for instance, the decision-making procedure used to initiate the package (discussed below). As explained above, the Genval criteria applied to operations before 2013 and the Joint Declaration to operations after that, with the following implications. ·The change of regime increased the possible amount of assistance. The Genval criteria set a ceiling for the amount of assistance at 60% of the residual financing gap for (pre-) accession and association countries and 33% of that gap for neighbourhood countries. On the insistence of the European Parliament, the 2013 Joint Declaration removed these ceilings whilst introducing the principles of fair burden-sharing and EU value added. However, the Commission has continued to apply the 60%/33% ceilings as a rule of thumb, since they were considered to adequately reflect the Declaration’s principle of fair burden-sharing. ·The 2013 Joint Declaration showed a clear preference for assistance in the form of loans (instead of grants), but the Genval criteria were more neutral in this regard. As Table 3 shows, this has resulted in a marked decrease in the share of grants over time. Procedures and decision-making In the absence of a framework regulation that would attribute powers to the Commission, operations have to be authorised by means of decisions – acts of secondary EU law, on the basis of Article 212 TFEU. Until 31 November 2009, the Council adopted such acts alone. Since then, the ordinary legislative procedure usually applies, with equal involvement of the European Parliament and the Council. The procedure follows the rules set out in Article 294 TFEU, as follows. The Commission triggers the procedure when it prepares and submits a proposal to the European Parliament and the Council, who reach their respective positions in ensuing internal deliberations before entering interinstitutional negotiations. In theory, the procedure could include a first and second reading in the Parliament and the Council, a conciliation procedure and, possibly, a third reading. However, in practice co-legislators usually reach early agreements during the first reading. When the situation in a third country requires urgent financial assistance from the EU, Article 213 TFUE allows the Council, acting alone in a non-legislative procedure, to adopt the necessary decisions on a proposal from the Commission. Since the adoption of the Joint Declaration, the Commission/ EEAS and its services must complete a certain number of steps to prepare the operation. On receiving a potential beneficiary’s request, they carry out an ex ante evaluation to assess the amount of financing needed and whether or not the country in question fulfils the eligibility conditions for assistance, including the political preconditions. The Commission then submits a proposal to the co-legislators. After the operation is authorised by the co-legislators, an MoU is negotiated and concluded between the beneficiary country and the European Commission, on behalf of the EU, setting out the reform conditions for the operation. The financial terms are set out in a loan agreement and, if the operation also comprises a grant, a grant agreement. The MoU is adopted as an implementing act under comitology, in line with the Joint Declaration. Figure 2: Process for granting and providing MFA The time lag between request and first disbursement varies a lot. In the 15 operations evaluated, the average length between the recipients’ request for assistance and the first disbursement was roughly 28 months and the median 26 months. However, since these figures include operations authorised before the entry into force of the Lisbon Treaty and during the protracted negotiations on the Commission proposal for an MFA Framework Regulation, putting some operations on hold, they do not give an accurate picture of the current state of play. This is why it is necessary to adjust the sample of operations in order to get a meaningful picture of the lags in the current decision-making procedures. Only 5 of the 15 operations evaluated took place within the current steady state (after the Joint Declaration entered into force in 2013). Excluding the special case of Ukraine (with three subsequent and sometimes overlapping operations), this would leave only the Tunisia and Jordan operations. It would therefore make more sense to describe the state of play on the basis of all operations since 2013. On the basis of such an adjusted sample, the following observations can be made (see Table 2 ). ·Whilst remaining somewhat variable, the average/median overall period between a request and the first disbursement was roughly 17 months. In some cases there was an improvement over time, as both the EU institutions and beneficiary countries became more familiar with the procedures. On the basis of more recent cases since 2020, it can be seen that the standard procedures make it possible for the period between a request and the first disbursement to be less than 10 months. ·Thanks to the use of urgency procedures in the European Parliament, the Commission proposal for the COVID-19 package was adopted in a record 1 month. The one-time use of Article 213 TFEU for urgent cases (Ukraine 2014) made it possible for only the Council to decide, also expediting the process. ·Entry into force of the MoU’s (Step III) sometimes requiring ratification by the beneficiary’s parliament was the main drag in the process, but the OLP (Step II) also sometimes added to the overall lag between a request and disbursement. Table 2: Time and steps (see Figure 3 ) from MFA request to (first) disbursement Operation↓ Duration in months Step I Step II Step III Step IV Total Albania 20 0 1 8 2 12 Bosnia and Herzegovina 20 0 1 12 4 18 Jordan 20-2 0 1 4 10 15 Kosovo 20 0 1 3 1 6 Moldova 20 0 1 4 2 7 Northern Macedonia 20 0 1 2 3 6 Montenegro 20 0 1 4 1 5 Tunisia 20 0 1 12 1 13 Ukraine 20 1 1 4 3 8 Jordan 20-1* 2 4 9 2 16 Ukraine 18 3 4 5 0 12 Moldova 17 17 8 4 21 50 Jordan 16 4 6 9 1 20 Tunisia 16 6 5 13 2 27 Ukraine 15 4 3 3 1 10 Ukraine 14 1 1 1 3 Tunisia 14 3 5 11 1 20 Jordan 13 5 8 5 9 26 Moldova 10 12 4 2 0 18 Ukraine 10 8 8 45 1 63 Median w/o COVID-MFA, w/o Art. 213 4 5 7 1 17 Average w/ COVID-MFA, w/o Art. 213 3 3 7 4 17

* Jordan 20-2 was a top-up of Jordan 20-1, part of the COVID-19 package and authorised in a multicountry (omnibus) decision. Amounts and financial terms The 15 operations evaluated enabled around EUR 4 billion of support to be made available on very advantageous terms. In total, roughly EUR 3.3 billion were disbursed to the beneficiaries. MFA most often takes the form of concessional medium-/long-term loans, the funds for which the Commission borrows on capital markets and lends on to the beneficiary country. MFA loans therefore had a typical maturity of around 15 years and were provided at the EU’s own borrowing costs over the same period – the costs of an AAA-rated debtor borrowing in their home currency. In some cases, MFA took the form of grants financed by the EU budget, or a combination of loans and grants, with the share of grants changing over time. Before the adoption of the 2013 Joint Declaration, MFA disbursements tended to have a significant share of grants. However, reflecting the Joint Declaration’s stated preference for loans and the limited budget available, since 2013 MFA grants have been provided only for eligible countries with very high vulnerabilities and comparably low income per head. The Commission is currently looking into whether or not the methodology for deciding on the provision of grants needs to be updated to maximise their usefulness while respecting the limited budget available in the 2021-2027 MFF. Table 3: Overview of financial terms of support provided Country

(in chronological order) Amount (EUR m)

authorised disbursed Share of grants Interest rate Maturity (months) Georgia 06 33.5 22 100% n/a n/a Kosovo 06 50 30 100 n/a n/a Moldova 07 45 45 100 n/a n/a Lebanon 07 80 40 38 3.1% 60 Georgia 09 46 46 50 177 Serbia 09 200 100 65 3.4% 96 Bosnia and

Herzegovina 09 100 100 0 2.0% 120 Moldova 10 90 90 100 n/a n/a Armenia 09 100 100 25 3.4% 179 Kyrgyz Republic 13 30 30 50 1.0% 176 Georgia 13 46 46 50 1.0% 175 Jordan 13 180 180 0 0.8% 176 Ukraine 10 and 14 1000 1000 0 1.4% 156 Tunisia 14 300 300 0 0.9% 161 Ukraine 15 1800 1200 0 0.7% 131 Total 4067 3329 Budgetary arrangements As a crisis instrument and unlike other EU external financing instruments, MFA does not provide financial support on a regular, programmable basis. When provided in the form of loans, its budgetary impact is limited to the provisioning of the External Action Guarantee in the Common Provisioning Fund financed from geographical budget lines under NDICI to cover the EU budget in the event of potential defaults. At a provisioning rate of 9%, this mechanism creates a significant multiplier between budgetary expenses and the headline amounts provided. To streamline its external financing architecture, the EU changed the budgetary set-up for MFA in 2021. In the 2021-2027 MFF, MFA grants continue to be budgeted separately (under budget line 14 20 03 01 Macro-financial assistance (MFA) – grants). However, the provisioning of the guarantee for MFA loans is now part of the budget allocated to NDICI in the External Action Guarantee, where the Regulation sets certain ceilings for the use of the budget for guarantees in general. Previously, MFA operations were provisioned in the Guarantee Fund for External Action (GFEA), which had a dedicated budget line. The assets constituted in the GFEA have now been transferred to the Common Provisioning Fund. In recent years, the EU has increased the budgetary means available for the instrument. Faced with increasing demand for MFA support in the context of the 2014-2020 mid-term review of the MFF, the Commission proposed (and co-legislators endorsed) an increase in the volume of lending capacity from EUR 0.5 billion to EUR 2 billion annually for the remainder of the programming period. Similarly, the Commission proposed that the volume of MFA lending remain consistent with these agreed yearly volumes – at EUR 14 billion – in preparation for the 2021-2027 MFF. During the NDICI negotiation process, the overall volume of guaranteed operations was reduced compared to the Commission proposal, with implications for the MFA budget as well. The adopted NDICI Regulation does not feature a specific envelope for MFA loans under the External Action Guarantee. However, as result of the overall ceiling for the external action guarantee established in the adopted NDICI Regulation, the maximum volume of MFA loans was fixed at EUR 11.6 billion. In addition, outside of the NDICI Regulation, a budget of close to EUR 350 million is envisaged for MFA grants. The budget for provisioning the External Action Guarantee is EUR 10 billion, covering European Fund for Sustainable Development+ operations, including MFA operations and Euratom loans granted during the period 2021-27. MFA conditions and compliance The 15 operations that form the basis of the meta-evaluation contained a total of 185 conditions, with compliance varying per operation as follows. Table 4: MFA conditions compliance rate by operation Operations↓ Fully compliant Partly compliant Not compliant Total of conditions Armenia 09 78% 22% 0% 9 Bosnia and Herzegovina 09 63% 38% 0% 8 Georgia 06 91% 9% 0% 11 Georgia 09 100% 0% 0% 6 Georgia 13 100% 0% 0% 8 Jordan 13 78% 11% 11% 9 Kosovo 06 100% 0% 0% 7 Kyrgyz Republic 13 100% 0% 0% 7 Lebanon 07* 29% 43% 29% 7 Moldova 07 100% 0% 0% 12 Moldova 10 70% 30% 0% 10 Serbia 09* 100% 0% 0% 9 Tunisia 14 67% 22% 11% 9 Ukraine 10 69% 24% 8% 26 Ukraine 14 82% 9% 9% 11 Ukraine 15 81% 6% 14% 36 * Given the non-disbursement of the second tranche, compliance was not officially recognised. The large majority of conditions covered budgetary reforms, including public financial management reforms. Almost a quarter of all conditions related to this policy area, followed by internal and external auditing (14%), social and healthcare policies (11%) and public procurement (10%). Chart 1: Distribution of MFA conditions across policy areas *Covers the financial management of the government, for example fiscal impact analysis, and everything related to the government budget. It also includes monetary policy and the organisation of the central bank, which is actually a separate topic but is taken into account here because there are too few reforms in this monetary policy area alone. Evaluations and methodology applied

The ex post evaluations of MFA have drawn on the 2010 and 2015 Guidelines for the Ex Post Evaluation of Macro-financial assistance Operations drawn up by DG ECFIN. The guidelines take account of the very specific nature of MFA operations and the resulting challenges for assessing them. The 2015 Guidelines ushered in a number of changes in the way the ‘core questions’ for MFA ex post evaluations were formulated. The 2010 Guidelines formulated the core questions in terms of how effective MFA had been in relation to macroeconomic stabilisation, supporting structural reform, and returning external debt to a sustainable path, as well as examining indirect/unexpected effects and how the MFA’s design affected its effectiveness or efficiency. The 2015 Guidelines adopted a different approach, focusing on the broader Better Regulation themes of relevance, efficiency, effectiveness, coherence and EU added value. Of the 15 evaluations that form the basis of this meta-evaluation, 8 were done using the 2010 Guidelines and 7 done using the 2015 Guidelines. Whilst all the ex post evaluations examined the contribution of MFA to short-term macroeconomic stabilisation, the impact on structural reforms, and various design and implementation issues, the precise mix of methodologies to do so varied from one evaluation to the next. In particular, there was a significant change between before and after 2015 in the evaluations’ suggested approach to counterfactual analysis. Until 2015, an essentially econometric modelling approach was suggested in the ECFIN Guidelines. However, it was judged that the economic models used for MFA did not produce reliable results. Since 2015, counterfactual analysis has therefore been carried out using relevant quantitative and qualitative techniques (Delphi technique and focus groups). 4.Evaluation findings (analytical part) 4.1 To what extent was the intervention successful and why? In line with the Better Regulations Guidelines, the performance of MFA should be analysed on the basis of its effectiveness, coherence and efficiency. Applied to MFA, ·“effectiveness” describes the instrument’s track record in meeting its stated objectives, i.e. contributing to macroeconomic stabilisation and promoting implementation of important structural reforms; ·“coherence” depicts the extent to which: oMFA operations support the Union’s own broader objectives in its external policies (directly and indirectly) -> “external coherence”, oa common framework of rules, policies and practices supports the instrument’s overall good and consistent functioning -> “internal coherence”: and ·“efficiency” relates to the ambition to deliver in the aforementioned dimensions at minimal costs to the overall resources available to the Union and the beneficiary country. Effectiveness in stabilising macroeconomic developments MFA has generally been effective in stabilising beneficiaries’ (external) macroeconomic environment, especially in the short and medium term. Besides the specific/individual assessments in the 15 evaluations previously mentioned, hard data back this statement up in the following areas. ·Despite a challenging external environment at the start of the respective MFA operations, market access could be maintained or eventually regained in all cases. This includes one beneficiary’s (Ukraine’s) restructuring its debt in good faith negotiations with private bondholders in 2015. With the exception of one strategic default (by Ukraine on a Eurobond of EUR 3 billion held by the Russian Federation that the Ukrainian authorities consider illegitimate), beneficiaries did not default. ·Reserve adequacy and current accounts mostly improved during MFA operations, and inflation subsided. Table 5: Selected macroeconomic indicators before/during/after MFA operations Current account balance (% GDP) Inflation (CPI) Foreign exchange reserves (in months of imports) Before MFA After Before MFA After Before MFA After Georgia 06 -8.1 -15.6 -12.3 9.5 9.4 5.8 1.7 3.1 3.2 Kosovo 06 -7.9 -9.2 -7.5 -0.7 3.1 3.9 4.6 2.9 2.5 Moldova 07 -7.9 -13.1 -0.3 12.3 12.5 5 3.0 5.1 2.1 Lebanon 07 -13.3 -16.8 -23.2 1.4 4.9 5.5 11.5 16.6 17.7 Bosnia and Herzegovina 09 -12.9 -7.2 -5.2 5 1.5 -1.2 4.5 5.9 7.3 Georgia 09 -19 -10.4 -10.2 9.5 4.4 2.3 1.7 3.1 3.2 Serbia 09 -15 -6.7 -7.4 9.7 8.5 8.7 4.7 7.0 5.2 Moldova 10 -7.9 -8.1 -5.7 8.3 6.5 6.4 7.9 5.2 5.2 Armenia 09 -14.8 -10.2 -5.8 6.6 5.1 4.2 4.4 4.5 4.2 Georgia 13 -12.3 -10.3 -6.5 4.9 2.9 4.2 3.3 3.2 3.2 Kyrgyzstan 13 -8.5 -14.6 -8.3 9.1 5.2 2 3.7 5.0 4.2 Jordan 13 -12 -8.1 -1.3 4.5 0.9 2.5 6.6 8.3 8.0 Tunisia I 14 -8.2 -9.8 0.4 4.4 4.5 6.6 3.2 3.2 3.8 Ukraine 10 and 14 -7.9 -1.1 -2.3 2.7 30.3 13.1 2.3 3.1 3.5 Ukraine 15 -7.1 -0.6 -0.5 4.1 25.7 7.3 Note. Before – average of the 3 years before MFA. MFA – average during MFA. After – average of the 3 years after MFA.

Source: IMF (April 2021), Worldbank These findings are more nuanced for timeframes longer than the medium term, based on the external study. Some improvements in current account balances are visible during that time, but gains in reserve adequacy could usually not be extended further and current accounts remained in deficit (a few years after the completion of MFA operations). Participants at the validation workshop tended to confirm this interpretation of the data.

However, these conclusions, especially insofar as they concern current account balances, need to be assessed carefully. There is no consensus among professional economists on the right current account position for a given country. Clearly, not all countries can reduce their current account deficits at the same time or consistently run current account surpluses. Moreover, for potential beneficiaries of MFA, universally in their economic transition, standard economic theory would suggest from a normative perspective that surpluses are undesirable, whilst the size of sustainable current account deficits depends on a variety of country- and time-specific factors. In any event, it must again be stressed that the beneficial short-to-medium-term effects outlined above cannot be attributed to MFA alone, given its relatively limited size and given its operations are undertaken in conjunction with IMF operations. In this vein, several stakeholders (including stakeholders at the validation workshop) underlined that concomitance with IMF operations has been essential in producing the results mentioned before. Interviewees from EU institutions have also pointed out that the complementarity of expertise and policy conditions has made the overall stabilisation operation more impactful. Interviews with stakeholders also indicate that cooperation between the EU, the IMF and the World Bank has increased political leverage for the EU to push for reforms. Table 6: Achievement of macroeconomic stabilisation according to the evaluation reports Effective for macroeconomic stabilisation If not, what was the justification for failure MFA size (% of current account deficit in the year before) MFA size (% of GDP in the year before) Was macroeconomic stabilisation achieved through the combined effect of MFA and IMF support? Armenia 09 Yes 11 1.4 Yes Bosnia and Herzegovina 09 Not much Small size 6 0.8 Yes Georgia 06 Yes 6 0.6 Yes Georgia 09 Not much Small size 2 0.5 Yes Georgia 13 Yes 3 0.4 Yes Jordan 13 Yes 7 0.7 Yes Kosovo 06 Yes 20 1.7 Yes Kyrgyz Republic 13 Yes 4 0.6 Yes Lebanon 07 Not much Small size 5 0.5 Yes Moldova 07 Yes 15 1.7 Yes Moldova 10 Marginal Small size 26 2.3 Yes Serbia 09 Marginal External context 2.8 0.6 Yes Tunisia 14 Not much Small size 9 0.9 Yes Ukraine 10 and 14 Yes 10 1.2 Yes Ukraine 15 Yes 10 1.0 Yes At the same time, it appears that concomitance with IMF support can at times result in implementation difficulties, such as disbursement delays. The external study says that interviewees and some OPC respondents expressed this concern. Some even advocated a review of the Joint Declaration’s Principle (6) on this requirement. Some stakeholders also argued that the scope of the EU’s leverage in an economic adjustment programme predefined by the IMF was not clear. In this context, the study reports that some stakeholders in IFIs wondered whether the EU could be more influential, also considering the amount of financial capacity there is in MFA. Others expressed concern that the close link to IMF support could be detrimental to the perception of the EU as an autonomous global actor. Effectiveness in incentivising reforms In contrast to the obligatory concomitance with IMF support, the link with political preconditions appears to be of less concern. In this regard, the external study focuses on one case (Moldova) in which their implementation caused some difficulties as the MFA operation was interrupted due to non-compliance. This in turn resulted in a desynchronisation with the continuing IMF operation. But otherwise, there is little reference in ex post evaluation reports to a (negative) impact of the political precondition on the outcomes, effectiveness and efficiency of MFA operations. OPC respondents also expressed little criticism in this regard, possibly because MFA is such an integral part of the EU’s external policy framework. For instance, the French senate confirmed this view in a 2017 report recommending a more prominent and explicit role for the political preconditions by including them, as well as the consequences of non-compliance, in the MoU (Marc, 2017). That said, the external study sees some room for improvement with regard to the way the political preconditions are assessed in practice. It seems that ensuring compliance is at times very challenging, for two main reasons. 1.Countries that benefit from MFA have different political systems. As a result, political preconditions might be more easily and swiftly fulfilled in some countries than in others. 2.As several stakeholders in EU institutions and IFIs acknowledged, the political preconditions are rarely specified. This makes it hard to assess their fulfilment, also because different Commission services/the EEAS may differ in their assessment. Like the link with political preconditions, the requirement to implement structural reforms enjoys relatively strong support among stakeholders. Furthermore, previous evaluations found that 81% of the requirements had been fully met, even if compliance rates differed between operations and targeted sectors. Despite this broadly strong track record, key stakeholders noted the prominent example of Ukraine (15), where excessive ambition led to a proliferation of requirements competing with each other in terms of implementation resources and political capital. The external study also identifies the 2014-2017 operation in Tunisia as having suffered from the same problem. Table 7: MFA conditions compliance rate by thematic groups Policy areas Total number of conditions Share of

fully compliant Share of

partly compliant Share of

not compliant Finance, Investment & Business environment 17 94% 6% 0% Public procurement 18 94% 6% 0% Energy sector 13 92% 8% 0% Social and Healthcare policies 20 85% 10% 5% Public Financial Management & Budget 42 83% 11% 5% Public Administration & Statistics 11 82% 18% 0% Privatisation & state-owned enterprises 5 80% 0% 20% Auditing (internal and external) 26 77% 20% 4% Trade & Competition 16 75% 13% 13% Anti-corruption 7 57% 14% 29% Taxation 10 40% 50% 10% Total 81% 14% 5% Given the high compliance rates, the external study finds little conclusive evidence of backtracking. What evidence is there is mainly in the area of the fight against corruption and in cases in which there had already been problems during implementation. These results are all the more noteworthy given previous analyses in this area. In 2007, the Independent Evaluation Office of the IMF found a compliance rate of just 54% in the IMF’s financial arrangements, with higher compliance by traditional interlocutors (finance ministries and central banks) in its core area of expertise (IEO, 2007). In comparison with this benchmark, MFA operations appear to be more successful in achieving implementation of the requested reforms. Several features support the effectiveness of MFA in incentivising meaningful reform based on the external study ·The instrument’s geographical focus played a supportive role. Interviewees suggested that the EU’s strong political ties with countries in its direct vicinity help ensure that the MFA conditions are met and that the instrument achieves its goals. Conversely, interviewees stated that fewer political and economic connections result in significantly lower leverage and that this potentially reduces the impact of the MFA. A majority of the OPC respondents (90%) likewise argued that Principle 2 of the Joint Declaration in its current formulation suits objectives of the MFA instrument either very well or rather well.There is some evidence that the high share of loans rather than grants can support the implementation of some types of reforms. This is underpinned by the theoretical notion that a beneficiary country has more incentive to implement revenue-raising reforms because they improve its repayment capacity. In this context, some shareholders support the Joint Declaration’s Principle 3 and its emphasis on loans. ·Similarly, there is broad support among interviewees from the EU institutions for the standard availability period of 2.5 years. This is considered sufficient to allow ambitious reform programmes to be implemented and has the benefit of allowing alignment with IMF arrangements (typically extending over 3 years). A shorter availability period could also mitigate the risk of a change of government and hence the loss of political will to implement a given reform programme, even if parliamentary ratification should to some extent help to avert such a scenario. ·The effectiveness of the MFA operations arguably stems at least in part from the political leverage they generate. This contributes to a more comprehensive reform effort that typically reinforces the effectiveness of the individual reforms undertaken.

Conversely, the external study also highlighted some features of the instrument that tend to reduce its effectiveness. ·The long procedure for authorising MFA operations undermines their effectiveness. As indicated in Section 3 above, the typical time lag between a recipient country’s request for assistance and the first disbursement was roughly 17 months, including 5 months for concluding the OLP approval process in the EU and sometimes including ratification by national parliaments. In particular, the time lag until the MoU (and therefore the MFA) enters into force may allow the economic situation to deteriorate further in the absence of full external funding support in cases where such a risk is typically mitigated by the timing of the underlying IMF assistance. Furthermore, the external study points to one case (i.e. Armenia 09), which was in urgent need of financial assistance and chose to seek front-loading from the IMF given MFA’s lengthy procedures. In this context, only half of the OPC respondents consider the procedure for granting MFA (Principle 7) to be very suitable. ·Notwithstanding the relatively high compliance rates, the MoU’s inflexibility appears to have reduced MFA’s effectiveness in times of change. MFA instalments are disbursed in full or not at all – unlike EU budget support, which is disbursed in increments as and when conditions are fulfilled. This is in line with IMF policies but, unlike those policies, MFA practice is that policy conditions are fixed at the operation’s inception and remain unchanged once the national parliament has ratified the text and the MFA has entered into force. The MoU may be amended upon mutual agreement, but the time that would be needed to complete the respective approval processes within the EU and the beneficiary country makes it almost impossible for the amendment to enter into force before the end of the availability period. Furthermore, in order to encourage a comprehensive approach to reform implementation, MFA does not allow for partial disbursements. Flexibility is therefore limited to waivers that exceptionally allow disbursements to take place despite non-fulfilment of one or several conditions, but new conditions cannot be added ‘ad hoc’ in order to reflect changes in the economic environment or a change caused by other drivers to the priority or assessment of necessary policy reforms. ·Assuming all else is equal, (small) volumes also affect effectiveness, as the case of Kosovo shows. The corresponding ex post evaluation report from 2012 stated that the overall contribution of MFA to macro-fiscal stability and structural reforms was limited due to a number of factors such as its volume. A total of EUR 30 million (0.6% of Kosovo’s GDP) was disbursed. Given the relatively small amount disbursed, the ‘real financial difference’ was moderate and so was the impact of the structural conditions linked to the MFA. The reference points for the amounts of the assistance (Principle 5) therefore appear to be somewhat controversial, with 70% of respondents judging them to be only ‘somewhat’ or ‘not very’ suitable. In any event, ownership of the agreed reforms is of critical importance for the MFA’s effectiveness in promoting important reforms. This is a consistent finding of analyses of the reform-based financial assistance and is also reflected in the external evaluation. Some factors (i.e. the lags in decision-making, and the relevance of reforms) therefore have an impact on the success of MFA operations through their impact on ownership. The external study similarly links cases where reform requirements lacked focus and were overly broad (Ukraine 10, 14) to a loss of ownership by the beneficiary country and therefore its compliance, which in turn reduces the MFA’s effectiveness in promoting reforms. Additional analysis is required in order to properly judge how the choice of policy conditionality affected the MFA’s effectiveness. The country-specific ex post evaluations as well as stakeholder consultation generally indicate that the policy areas covered by the MFA conditions are relevant. However, the methodology for assessing whether the reform areas chosen were well-targeted, well-sequenced and sufficiently SMART (Specific, Measurable, Attainable, Realistic and Time-bound) to support the MFA’s overall objectives needs to be refined. Such analysis would not be straightforward, however, given the wide scope of the economic and structural MFA objectives on the one hand and the instrument’s contribution to the EU’s broader objectives (as reflected, for example, in the political preconditions) on the other. Further efforts to explore this issue could be useful. Efficiency: The external evaluation confirms that MFA is a relatively (cost-)efficient instrument for the EU. As explained above and in line with the provisioning policies of the External Action Guarantee (Article 31, NDICIR) and its predecessor the External Action Guarantee Fund, provisions are set for MFA loans at 9% of the principal amount to be disbursed. Given that no beneficiary has defaulted to date on an MFA loan, the related external guarantee provisioning has been routinely returned to the EU budget. Some of the principles mentioned above further support this high cost-efficiency. Concomitance with IMF interventions provides some assurance concerning the country’s repayment capacity (European Commission, 2021) because the IMF does not usually provide funding to countries with an unsustainable debt (unless it is taking measures to address this issue). In addition, the synergies arising from relying on the IMF’s work generate important cost-savings on the EU side. The geographical focus similarly creates synergies in terms of expertise and human resources between and within different Commission departments. MFA is also cost-efficient for the beneficiary countries because its loans are highly concessional. The EU’s high credit rating allows it to borrow money at much lower interest rates than beneficiary countries. The EU passes on these interest rates in its loan agreements, so the beneficiary countries receive a financial benefit that is often substantial. This benefit can be expressed as the difference between the nominal value of a credit (on the basis of market rates) and the present value of the debt service on the date of disbursement (calculated at a discount rate applicable to the currency of the transaction and expressed as a percentage of the nominal). Table 8: Concessionality of selected operations under the COVID-19 MFA package* Country Interest rate Amount (EUR) Concessionality (%) Kosovo 0.125% 50 000 000 61.8% North Macedonia 0.125% 80 000 000 56.3% Montenegro 0.125% 30 000 000 56.3% Jordan 0.125% 150 000 000 61.5% Georgia 0.125% 75 000 000 61.5% Moldova 0.125% 50 000 000 61.5% Ukraine 0.125% 600 000 000 61.4% * The COVID-19 MFAs are cited as an example here because they consisted entirely of loans. The current EU decision-making arrangements can cause some inefficiencies. By reaching agreement on a template for MFA decisions as a follow-up to the 2013 Joint Declaration, the EU’s institutions have revealed relatively little divergence in terms of the substance of MFA operations during the OLP approval negotiations. In the case of MFA, the main purpose of legislative negotiations (to resolve disagreement) was therefore largely absent, but the co-legislators were still required to go through the process required for a standard OLP negotiation. The co-legislators generally needed 5 months to adopt the Commission’s proposal for MFA operations under review. In addition and before MFA enters into force, the MoU has to be formally agreed (see Step III in Table 2 ) – which requires a Commission implementing act subject to Comitology (which is also generally used to decide on the EU’s budget support outside of emergencies). It took roughly 7 months to conclude the negotiations with the beneficiary country and adopt the MoU, but this can take less time if the negotiations are conducted largely in parallel with the OLP process. Besides, recent experience with the COVID-19 MFA package and with MFA support to Ukraine in the context of Russia’s unjustified invasion of the country has shown that MFA’s current set-up can allow very swift adoption if all institutions agree to make full use of the available procedural flexibilities. Questions can also be raised over the efficiency of the decision-making process in beneficiary countries, which often also ratify the MoUs. This can create a stronger sense of ownership of the reforms to be implemented, but tends to delay the disbursement of the MFA and all but prevents MoUs’ subsequent adaptation (as discussed above). In addition, national ratification is not common practice for IMF programme documents or the conditions attached to the EU’s budget support under ENPI/NDICI or IPA III. Coherence: 1.Coherence with other EU instruments As indicated above, MFA usually intervenes in tandem with other EU instruments, such as currently NDICI and IPA. This results in significant synergies, chiefly with budget support under these instruments. As Table 7 shows, MFA conditions often include reforms related to PFM/budget and auditing. These areas are also critical to the effectiveness of the budget support operations under these instruments, where the MFA operation requires implementation of all conditions before disbursement can take place, thus strengthening the incentives to carry out comprehensive reforms in these areas. MFA conditions therefore tend to be closely dovetailed with budget support conditions so that they can reinforce and complement them as much as possible.

In order to underpin such coherence with other instruments, coordination between Commission services (DG ECFIN, DG NEAR, DG INTPA and others) takes place on an ongoing basis. This reflects the need to address specific issues related to the MFA conditions, insofar as they are under the responsibility of the respective DGs. The co-existence of intervention through MFA with intervention through other instruments such as ENPI has nevertheless produced some issues at times. For instance, the ex post evaluation of Moldova 07 found that it had resulted in modest additional administrative costs because two steering committees and two reporting mechanisms were run at the same time. It also found that MFA and ENPI policy conditions had overlapped, and that this had resulted in some synergies but also had caused some ‘confusion’. The ex post evaluation does not specify the consequences of this confusion, but the external study carried out for this meta-evaluation argues that the overlap of conditions probably made it harder for both the EU and the beneficiary country to keep track of the reforms. By contrast, the example of Armenia stands out as an illustration of how a high level of coherence can be achieved and make a real difference. According to the corresponding evaluation, the MoU sought to ensure consistency with the policy lines of the IMF-supported economic adjustment programme and the priorities of the ENPI and the Eastern Partnership. Alongside the ENPI, MFA conditions targeted socio-economic reforms and sustainable development, and supported good economic governance, efficient public spending, trade and investment. Reforms were selected which were likely to be achievable during the MFA operation and which non-governmental national and international stakeholders also identified as priority issues. In fact, those reforms also prepared the country to meet preconditions for the DCFTA negotiations between May 2012 and July 2013. All in all, MFA’s coherence with other EU instruments appears to be good. ·As the stakeholders confirmed, the objectives, design and implementation of MFA are complementary and consistent with other initiatives providing financial support to beneficiary countries. MFA complements the budget support programmes in the beneficiary countries. The two instruments’ conditions are aligned as far as possible. ·There appears to be some potential for the MFA to provide additional benefit to other EU external policies, chiefly in conjunction with budget support operations. It should nevertheless be noted that there are limits to the level of coherence that the MFA can deliver without losing its stand-alone rationale and nature. As the Joint Declaration explicitly states, MFA is not an instrument for long-term (‘development’) financing. It has a specific place and role in the EU’s external policy toolkit of focusing on relatively short-term and repayable balance of payment support with macroeconomic significance. Coherence must therefore not be understood as replicating the intervention model of budget support because that might compromise the complementarity of those instruments with each other. 2.Coherence with the interventions of international partners Another dimension of the MFA’s coherence with other instruments concerns coordination between the Commission and the IMF. Synchronisation with the parallel IMF reviews was sometimes challenging and some conditions overlapped. That said, these coordination difficulties did not undermine the overall success of MFA . Close coordination with the IMF and occasional interlinking with interventions by the World Bank enabled the Commission to draw on the expertise of these organisations. Coordination has therefore been reported by the stakeholders as ‘good’ between MFA, other EU support, the IMF and other sponsors. Complementarity between MFA and IMF support resulted in a joint impact that exceeded what both initiatives could have achieved alone. The stakeholder consultation shows that external coherence with IFI interventions is generally seen as high and to be supporting the operation of the instrument. Stakeholders mainly referred to complementarity and the absence of overlap between the conditions of the different actors once their different interests, mandates and approaches had been taken into account. Regarding internal coherence, stakeholder feedback suggests that MFA has been consistently and clearly developed, so that its implementation follows the same set of principles and rules in all beneficiary countries. This internal coherence is founded on the Joint Declaration, which provides a common framework despite the lack of a dedicated legal act. However, when assessing Better Regulation criteria, the external study finds some variation in the outcome of individual operations. This variation may reflect specific country and/or time factors, but could also indicate that further standardisation work is needed. When effectiveness, coherence and efficiency are all taken into account, the overall assessment of the MFA instrument’s success over the last decade is positive. The analysed evidence suggests that MFA helped stabilise beneficiaries’ economies and promoted the implementation of important policy reforms. The instrument’s external and internal coherence was satisfactory overall, even if external coherence with other EU external policy instruments could be further enhanced. Finally, MFA is very cost-efficient, especially when support is provided in the form of loans, although swifter upstream decision-making procedures could produce additional resource savings. 4.2 How did the EU intervention make a difference? From the Member States’ perspective, EU MFA operations provide financial and political value added. The financing of MFA loans by means of EU borrowing does not affect Member States’ finances or debt ratios, but does allow them to provide support in regions that they have interest in. Consistency with the EU’s external policies allows more economies of scope (within the Commission and the EEAS) than smaller Member States in particular would otherwise be able to mobilise. This added value tends to increase with the volume of the operation. From the beneficiaries’ perspective, all the consulted stakeholders recognised the value added by MFA. According to them, a majority of the reforms implemented as part of MFA’s reform requirements would not have taken place otherwise (or at least not as quickly) – e.g. reforms in the areas of public financial management, anti-corruption and the judicial system. This is despite the fact that the scope and overall impact of the MFA operations in their countries were relatively small compared with the overall EU presence there. As some key stakeholders suggest, MFA supports the EU’s unique role in fostering reforms to strengthen the rule of law in particular. Interviewed stakeholders from both the Commission and the IFIs suggested that not all the MFA conditions were directly affecting macro-economic stability in the beneficiary countries. Stakeholders considered this to be an advantage, however, especially because MFA in the beneficiary countries works alongside the IMF and other EU instruments. Unlike the IMF, which is committed to strict neutrality, MFA promotes democratic values directly as well as other values that are of interest to the EU and the beneficiary countries. Rule of law and democratic values are not MFA operational objectives, however, but are rather pre-conditions for obtaining MFA. This is despite the observation that the size of EU added value is perceived differently across operations, both by external evaluators and by beneficiary countries. According to the external study, this may reflect different levels of involvement by the EU (compared with the IMF and the World Bank) in the perception of both the external evaluators and the beneficiary countries. This could be because (i) IMF support has to be in place before the authorisation of an MFA and (ii) according to interviews with the IFIs, the IMF was usually ‘making a bit more noise’ about its support than the EU.

The interviewed stakeholders have noted in this context that MFA’s visibility was rather low in most beneficiary countries. This low visibility reduced the value added by the MFA because the instrument was under-utilised for the promotion of EU economic diplomacy in beneficiary countries and their geographical surroundings. The meta-evaluation finds that the EU could increase the value it adds through more active engagement with political leaders, greater media coverage and engagement with a wider group of stakeholders, including the private sector and civil society. Some stakeholders also mentioned the possibility of communicating more consistently on the whole package of the EU’s assistance, including MFA. Some key stakeholders were convinced that increased visibility can help the EU to reach out to non-traditional interlocutors and thus provide new opportunities to form ‘coalitions for reform’, increase the sense of ownership in the beneficiary countries, and thus make MFA more effective in promoting important reforms. 4.3 Is the intervention still relevant? Continuing demand for MFA operations from beneficiaries demonstrates the instrument’s ongoing relevance. During the evaluation period, MFA was requested and deployed 15 times. Moreover, 10 operations were requested and authorised at the inception of the COVID-19 pandemic in 2020. The Commission received further requests by the end of 2021, despite the increased availability of global liquidity given the 2021 general allocation of special drawing rights (SDRs) by the IMF. The emergency and exceptional MFA provided to Ukraine in 2022 in the context of Russia’s unjustified war of aggression is further important testimony of the usefulness of MFA as a flexible crisis instrument. On the basis of past operations, the external study also draws a positive conclusion on the relevance of policy reforms promoted through MFA. MFA had an operational and/or political reinforcing effect in all areas in which conditions were formulated (Moldova). It often therefore acted as a positive contributing factor rather than the sole cause of a reform (Ukraine 10.14). MFA similarly promoted actions that might not have been implemented otherwise while, in some other instances, the authorities would probably have taken action even without the MFA conditions (e.g. Moldova). In terms of scope, they addressed important issues, even if they did not always comprehensively cover all of the most pressing reform needs (e.g. Bosnia and Herzegovina). Reforms in Public Financial Management (PFM) and the social safety net, which are often the subject of conditions (see Table 7 ) were particularly relevant (Ukraine 10.14). Against this backdrop, the external study concludes that the MFA instrument has a high level of relevance in most countries. Structural reform conditions enshrined in the MoU are relevant and most of them have been implemented as agreed. Only in a few cases was implementation of the agreed reforms slow and ownership of the reform programme limited (e.g. Tunisia 14). 4.4 Is the current framework for evaluating interventions appropriate? With the 2015 ‘Guidelines for the Ex-Post Evaluation of Macro-Financial Assistance Operations’, the MFA ex post evaluations started to follow the Commission’s Better Regulation Guidelines (BRG). Previous evaluations did examine the five criteria of effectiveness, efficiency, coherence, relevance and EU added value but the BRG provide more detailed guidance on the general approach to be used. More specifically, BRG brought them together, gave them a more precise and standardised definition, and placed them at the heart of the evaluation framework for EU-supported interventions including the MFA operations. The post-BRG ex post evaluation reports were therefore structured more closely around the five criteria after 2015 than had previously been the case according to the external study. The evidence demonstrates that the MFA evaluations throughout the period generally provided a good assessment of the key evaluation criteria set out in the 2015 BRG. These key criteria featured in the MFA ex post evaluations throughout the period, albeit to a varying degree. Within the overall BRG framework, each evaluation was meant to be tailored to the objectives of the specific programme and its expected outputs, results and impact. Whereas earlier, pre-2015 ex post evaluation reports had been structured around the main MFA objectives (macroeconomic impacts, contribution to structural reforms, and the design and implementation of the MFA operations), the later post-2015 ex post evaluation reports were largely structured around the five evaluation criteria. In addition to the five evaluation criteria, the BRG encouraged a more in-depth assessment of social impacts than had previously been evident. The Social Impact Analysis (SIA) included in the MFA ex post evaluations was intended to improve the understanding of the extent to which MFA interventions, which primarily had an economic objective, also had effects – intended or otherwise - of a social nature and whether these were positive or negative, or a combination of these effects depending on geographical, social and other factors. This was a new type of analysis. In contrast, to the Debt Sustainability Analysis (DSA) introduced by the BRG was less novel to the ex post evaluations, because it was essentially an extension of the existing macroeconomic impact assessment. Overall, the use of SIAs and DSAs made a significant contribution to the assessment of MFA efficiency and effectiveness, based on the external study. With the DSAs the difference between the pre- and post-BRG level of analysis was less pronounced because this type of analysis was already a key element in the earlier ex post evaluations. The DSAs improved the understanding not only of the macroeconomic impacts – which were central to all the ex post evaluations, whether pre- or post-BRG – but also of the longer-terms effects in terms of debt sustainability. The BRG also placed increased emphasis on the systemic involvement of key stakeholders (both in terms of impact assessments and ex post evaluations). There had already been quite extensive consultations with stakeholders in Brussels, the beneficiary countries and the international financial institutions as part of the earlier MFA ex post evaluations. However, the 2015 BRG encouraged evaluators to further develop this aspect of their research, with examples including the Delphi technique successfully used to a greater extent in engaging a wide range of stakeholders with a knowledge of MFA. The 2015 DG ECFIN evaluation guidelines enshrined this principle further by de-emphasising econometric modelling for the purpose of counterfactual analysis. The analysis of high-frequency economic indicators has instead been supplemented by greater stakeholder consultation in the construction of counterfactuals. This decision was in line with a 2012 CEPS-report commissioned by the European Parliament which concluded the following on this matter: ‘The meta-evaluation report [of 2010] devoted attention to the question whether econometric modelling techniques could materially help the evaluation process. Their conclusion, which this reviewer shares, is that there are both practical and theoretical reasons why this could add little value to the evaluations.’ The methodological approach to MFA counterfactual modelling is in line with how DG ECFIN has evaluated euro area economic adjustment programmes where a crisis makes counterfactual scenarios underpinned by econometric modelling less reliable. Such an approach also has parallels in the work of other institutions (e.g. the IMF’s Independent Evaluation Office (IEO) recognises ‘significant empirical challenges involved in identifying appropriate counterfactuals and isolating the impact of programmes’ ). When undertaking individual MFA evaluations, DG ECFIN does not prescriptively set an explicit methodology to be followed by external contractors. Instead, an overarching set of MFA evaluation methodological guidelines complements the Better Regulation requirements. This promotes competition and innovation among external contractors in devising suitable evaluation methodology, not just for counterfactual analysis but also for the overall assessment of the evaluation criteria. Examples include (i) analysis of fiscal savings relating to favourable debt terms used by one contractor that has subsequently been applied in other MFA evaluations, and (ii) the innovative use of social media analysis by a contractor in the assessment of relevance, effectiveness, efficiency and EU added value in the evaluation of MFA Ukraine I & II. Against this backdrop, it seems that the standard of individual evaluations was good and therefore that, by extension, the reliability and relevance of the methodologies employed was appropriate. Evaluations followed the relevant methodological guidelines and were further enhanced by the introduction of the BRG into the MFA evaluation framework. The evaluations provide comprehensive and detailed insights into the MFA’s macroeconomic impacts, its contribution to structural reforms and other issues relating to the design and implementation of the MFA operations. In particular, it can be concluded that most of the MFA evaluations provided a good assessment of what would have happened without the MFA operations. The counterfactual analysis typically focused on macroeconomic issues but was also extended in a number of cases to the assessment of the MFAs’ contribution to structural reforms. This overall positive assessment notwithstanding, the external evaluator has also found several points for improvement in the evaluation of the individual MFA programmes in the future: 1.The comparative dimension of the ex post evaluations was a relatively weak aspect of the assessments and ‘lessons learned’ were therefore not a prominent feature. More specifically, only two evaluations had dedicated sections on this matter, whereas around two thirds of the MFA ex post evaluation reports included ‘recommendations’. This was particularly the case in pre-2015/pre-BRG evaluations. 2.The assessment of coherence can be further improved by placing more emphasis on assessing all the EU-supported programmes as a whole in a particular beneficiary country. In addition, elements in MFA operations that contribute directly or indirectly to wider EU policy goals could be more clearly highlighted. 3.The issue of low MFA visibility also extends to its ex post evaluations. The evaluations’ findings were usually not disseminated systematically after the reports were finalised. Some stakeholders such as central banks, ministries of finances, IMF/World Bank did at times have access to the information because they participated in validation workshops, but this was not the case for other stakeholders such as local chambers of commerce and industry, some IFIs (e.g. the EBRD), representatives of civil society, academics, and the business and financial sector.

5.What are the conclusions and lessons learned? 5.1 Conclusions The overall assessment of the MFA instrument over the meta-evaluation period is very positive. It contributed to operations that stabilised the economy of beneficiaries. It was very cost-efficient and provided an appropriate level of coherence with other EU instruments in neighbourhood and pre-accession countries. Continuing demand confirms its ongoing relevance and stakeholders widely confirm the EU value added of past operations. Furthermore, the framework for MFA evaluations is assessed as appropriate and useful. Ex post evaluations were carried out to a high standard and were further enhanced with the introduction of the BRG into the MFA evaluation framework. More specifically, the BRG encouraged a more in-depth assessment of social impacts and a better appreciation of the five evaluation criteria of effectiveness, efficiency, coherence, EU value added and relevance. The evaluations have therefore provided comprehensive and detailed insights into MFA’s macroeconomic impact; its contribution to better governance, more economic resilience and/or sustainable growth models through structural reforms; and other issues relating to the design and implementation of the instrument. Most MFA evaluations have therefore provided a good indication of what would have happened without the MFA operations. Several points for improvement have nonetheless been identified. 1.Concomitance with IMF support: concomitance with IMF support significantly improved MFA’s cost-efficiency and effectiveness, mitigating financial risk and allowing division of labour. At the same time, making the MFA strictly conditional on a satisfactory track record of implementing an IMF-supported programme has sometimes caused implementation delays at the start of and during operations, and this has prompted some stakeholders to call for more flexibility. Furthermore, some are concerned that the standard arrangements with regard to IMF involvement could compromise the EU’s standing as an autonomous global actor. A more flexible approach was employed in this respect in recent exceptional cases (the COVID-19 MFA package and 2022 MFA to Ukraine). While clearly exceptional, this experience could, once evaluated, usefully feed into further reflections on the pros and cons of potential alternatives to the current regime. 2.Decision-making procedure: the current standard procedures in the EU (the OLP) and the beneficiary country (parliamentary ratification) have been identified as a potential source of inefficiency and risk, thus reducing the instrument’s effectiveness in addressing an acute or impending balance of payments crisis and promoting reform implementation within the standard time window of IMF-supported economic adjustment programmes. It was the intention of the Commission in 2011 to establish a framework regulation for MFA that would make the instrument more effective by streamlining its decision-making process, but no agreement could be reached on a legal set-up that was acceptable to all EU institutions. Recent experiences with the COVID-19 MFA package and with MFA support to Ukraine in the context of Russia’s unjustified invasion of the country have shown that the current set-up of MFA decision-making can allow very swift adoption if all institutions agree to make full use of the available procedural flexibilities in particularly urgent cases. 3.Volume of financial assistance: although critically dependent on the design of its policy conditions, the impact of MFA operations (and hence their effectiveness and added value) is (if all other factors are the same) proportional to the level of the financial support. The 2013 Joint Declaration no longer specifies specific ceilings, but it is not yet fully clear to what extent the current policies provide sufficient flexibility. This issue may therefore need to be revisited at a later stage. 4.Policy conditions: policy conditions that are appropriate for the chosen policy areas, their degree of ambition and impact on long-term sustainability, as well as their sequencing over time and focus are important elements of a successful MFA operation. Individual ex post evaluations have generally been positive regarding the relevance and impact of MFA reform conditionality. 5.Flexibility: the inability to adapt the conditions listed in the MoU distinguishes MFA from IMF support and the lack of adaptable disbursement modalities distinguishes it from EU budget support. A more flexible, IMF-like regime appears to have the advantage of being able to effectively anchor and accompany important policy reforms, especially in a fast-moving economic situation which could influence decisions on the most suitable policy response and therefore the necessary reform commitments. In the context of MFA, some flexibility is provided through a limited number of waivers which exceptionally allow disbursements to take place despite the non-fulfilment of one (or a few) reform conditions. Furthermore, in the 2022 exceptional MFA package for Ukraine, allowances for the uniquely challenging war situation were made in advance when designing conditionality in the MoU. The increased flexibility enabling the use of MFA conditionality even in this context proves that, while recognising the exceptional character of these operations, solid reform conditionality remains a cornerstone of successful MFA operations. 6.Visibility of MFA operations and their evaluations: MFA’s low visibility meant that the instrument did not realise its full potential to promote EU ‘economic diplomacy’ in beneficiary countries. More specifically, greater visibility would make it possible to reach out to non-traditional interlocutors and thus create opportunities to form ‘coalitions for reform’ to increase a sense of ownership in the beneficiary countries and thus support the MFA’s effectiveness in promoting important reforms. 5.2 Lessons learned The above-mentioned conclusions provide a basis for further reflection, but there are also a few ‘lessons learned’ which the Commission could consider acting upon in the future. 1.Political pre-conditions: as explained in Section 4.1, the external study finds some challenges in the way political pre-conditions are assessed in practice that could have a bearing on MFA’s effectiveness. The requirements for continued MFA support should therefore be communicated more clearly in cases where actual or possible disruptions emerge in relation to the partner country’s respect of effective democratic mechanisms, including a multi-party parliamentary system, the rule of law and human rights. 2.Decision-making: Like the external meta-evaluation of 2009/10, the current one points to the risks brought to the instrument’s effectiveness and efficiency from the standard decision-making procedures at the side of the EU and in the beneficiary country. Although the EU decision-making was greatly expedited in the most recent COVID-19-related operations as well as for the emergency and exceptional MFA to Ukraine, the use of urgency procedures, notably by the European Parliament, is fully within each institution’s prerogatives and hence highly uncertain as to its use in future cases. At this stage, however, there is no clear indication given that changing the legal set-up would significantly speed up MFA operations beyond what can be achieved when making full use of the available procedural flexibilities. Further reflections may be useful on potential alternatives to the current decision-making procedure. These aspects could usefully be assessed once the evaluations of the COVID-19 MFA package and the exceptional Ukraine MFA package have been carried out. 3.MFA visibility and dissemination of MFA evaluations’ findings: the low visibility of MFA operations is an important finding of the evaluation and touches on MFA outcomes in several dimensions (effectiveness, coherence, and EU added value). Possible improvements in this regard should therefore be explored, including with regard to the dissemination of ex post evaluation results. 4.‘Lessons learned’: future MFA evaluations could place more emphasis on ‘lessons learned’. This is in line with the November 2021 updates to the Better Regulation Guidelines and could be made a specific requirement in the terms of reference for final ex post evaluation reports. More emphasis on ‘lessons learned’ and methods of sharing these with key stakeholders would enhance the benefits of future evaluation activities.



Annex I: Procedural Information

In 2020, the Commission’s DG ECFIN took the initiative for a new meta-evaluation covering 15 MFA evaluations completed in the last ten years. The exercise, coinciding with the MFA instrument’s 30th anniversary, takes stock of previous evaluation results and aims to support organisational learning both in terms of the design and implementation of the MFA instrument and the methodology used to evaluate individual operations. The meta-evaluation also aims to support transparency and accountability and to contribute to laying the groundwork for any possible improvements of the instrument’s design and/or workings as necessary. The Decide planning entry for the meta-evaluation is PLAN/2020/6586. In line with the requirements set out in the Commission Better Regulation Guidelines, the meta-evaluation considered the criteria of relevance, effectiveness, efficiency, EU value added, coherence with other international and Union instruments for financial assistance, social impact and impact on the sustainability of public debts. The exercise relies primarily on an external independent study commissioned by the Commission in December 2020 and concluded in October 2021. In order to ensure validity, the analysis and conclusions of the external study are based on the evidence obtained using several evaluation methods (document review, semi-structured interviews with targeted stakeholders, open public consultation, a validation workshop and a roundtable with relevant Commission staff). A number of limitations were experienced during the preparation and completion of the study. Annex 2 provides a detailed overview of these shortcomings, together with the mitigation strategies adopted. The outcome of the external assessment was supplemented by input from (i) a high-level seminar discussing the experiences of 30 years of MFA that took place in November 2021 (see below for a report), and (ii) the Regulatory Scrutiny Board (RSB). The lead DG to carry out and manage the meta-evaluation has been DG ECFIN. DG ECFIN also chaired the inter-service group (ISG) that was set up to manage the external evaluation. Apart from DG ECFIN, the ISG comprised of representatives of other Commission services (the Secretariat General, the Directorate-General for Neighbourhood and Enlargement Negotiations and the Directorate-General for International Partnerships). The ISG had the responsibility to: a.Establish the evaluation roadmap for the external evaluation; b.Establish the Terms of Reference (ToR); c.Facilitate the evaluator’s access to the information needed; d.Advise, monitor and comment on the work undertaken by the external contractor. The call for tenders was launched in October 2020 and two applications were received. Following the verdict of an evaluation committee, the specific contract to undertake the external evaluation was awarded to a consortium led by OXFORD RESEARCH under Framework Contract number ECFIN-003-2017 - ref. Ares(2017)1628394. The consortium included VVA Brussels and the Centre for Strategy & Evaluation Services. The total amount paid under this specific contract was EUR 135,200.00 covering all tasks executed. A kick-off meeting, where the ISG and the external contractor discussed the deliverables and the evaluation methods, took place in December 2020. The evaluation roadmap ran between 21 January 2021 and 18 February 2021, to seek wider feedback. This was followed by the launch of the OPC on 30 March, for a period of twelve weeks (OPC Decide planning entry PLAN/2020/6586). Meetings on the inception and interim reports were held in, respectively, January 2021 and April 2021. In order to provide appropriate support and ensure the evaluator's access to the necessary information as comprehensively as possible, bi-weekly dialogues were organised between the evaluator and DG ECFIN from May 2021 onwards. These dialogues continued until the end of the exercise. To address identified knowledge gaps on the MFA instrument noted in the interim report, ISG members also set up a dialogue between the ex