Table of contents Glossary 1.Introduction Purpose of the evaluation Scope of the evaluation 2.Background to the intervention Description of the intervention and its objectives Points of comparison 3.Implementation / state of play Economic situation Implementation of the MFA operation 4.Methodology Evaluation techniques used Risks and limitations 5.Analysis and answers to the evaluation questions Evaluation Question 1: Relevance of the operation Evaluation Question 2: Effectiveness Evaluation Question 3: Efficiency of the operation Evaluation Question 4: EU added-value of the operation

Evaluation Question 5: Coherence of the operation Evaluation Question 6: Social impact of the operation Evaluation Question 7: Public Debt Sustainability of the operation 6.Conclusions Annex 1: Procedural information Organisation, design and timing Exceptions to the Better Regulation Guidelines Annex 2: Stakeholder consultation, including summary of results Annex 3: Timeline of the MFA operation Glossary Term Meaning or definition DCFTA Deep and Comprehensive Free Trade Area DG ECFIN Directorate-General for Economic and Financial Affairs DSA Debt Sustainability Analysis EC European Commission EFF Extended Fund Facility EIB European Investment Bank EMBI JPMorgan Emerging Market Bond Index

EQ Evaluation question EU European Union EUR Euro FDI Foreign Direct Investment FSAP Financial Sector Assessment Programme FX Foreign exchange GDP Gross Domestic Product GEL Georgian lari ICAAP Internal Capital Adequacy Assessment Process IFI International Financial Institution IMF International Monetary Fund ISG Inter-Service Steering Group LHS Left hand scale MFA Macro-financial assistance MOF Ministry of Finance MoU Memorandum of Understanding NBG National Bank of Georgia NGO Non-governmental organisation PFM Public Finance Management PPP Public Private Partnership

RHS Right Hand Scale SAO State Audit Office SBA Stand-By Agreement SCF Stand-By Credit Facility SREP Supervisory Review and Evaluation Process SWD Staff Working Document ToR Terms of Reference UHC Universal Healthcare US United States USD US dollar 1.Introduction Purpose of the evaluation This Staff Working Document presents an evaluation of Macro-Financial Assistance (MFA) to Georgia, which was implemented in 2015-2017 (MFA operation of 2015-2017). The evaluation is based on a study undertaken by an external contractor and a consultation process that involved various stakeholders, as well as European Union (EU) staff working with Georgia. The aim of the evaluation is to assess whether the MFA operation of 2015-2017 met its objectives to support Georgia in addressing its balance-of-payments problems and implementing economic and structural reforms that would stabilise its economy and enhance the sustainability of its external position. The purpose of the evaluation is to inform the future use of the MFA instrument, while also ensuring transparency and accountability. Scope of the evaluation The MFA instrument is a form of undesignated financial aid extended by the EU to partner countries experiencing a balance-of-payments crisis. MFA takes the form of medium/long-term loans, grants or a combination of the two. It is designed for third countries that are geographically, economically and politically close to the EU. Unlike other, regular development aid provided by the EU, MFA is exceptional in nature and is mobilised on a case-by-case basis. Its objective is to restore a sustainable external financial situation, while encouraging economic adjustment and structural reforms. MFA generally complements financing provided in the context of a reform programme agreed with the International Monetary Fund (IMF).

In accordance with the Financial Regulation (Article 30(4)), MFA operations in third countries are subject to an ex-post evaluation. In turn, the aforementioned MFA decision for Georgia stipulates that the European Commission is required to “submit to the European Parliament and to the Council an ex post evaluation report.” To this end, DG ECFIN hired an external contractor to complete an evaluation report, which informs this Staff Working Document (SWD). The external evaluation ran from September 2018 until April 2019. The objective of this ex-post evaluation is twofold: 1.To analyse the impact of the MFA on the economy of Georgia, and in particular, on the sustainability of its external position; and 2.To assess the added value of the EU’s intervention. In general, the evaluation seeks to draw lessons with respect to the EU’s financial assistance, i.e. a.Whether the ex-ante considerations determining the design and terms of the operation were appropriate, taking due account of the economic, political and institutional context; and b.Whether the outcome of the programme met the objectives. These main areas are assessed along the following key evaluation criteria: relevance, effectiveness, efficiency, EU added-value, and coherence with other EU policies. In addition, the evaluation assesses the social impact of the MFA and the impact on the sustainability of Georgia’s public debt. 2.Background to the intervention Description of the intervention and its objectives As described in detail in chapters 2 and 5 of the external evaluation report annexed to this SWD, Georgia’s economy was heavily hit by an armed conflict with Russia in August 2008 and the global financial crisis. These severe external shocks, preceded by Russian restrictions on some Georgian exports from 2006, aggravated the existing internal imbalances and necessitated international financial support to Georgia. In the external sector, Georgia’s trade balance worsened due to weaker export performance, as both the demand for exports and commodity prices decreased in the wake of the aforementioned conflict with Russia and the global financial crisis. While the initial impact on the trade balance was to some extent mitigated by an abrupt adjustment (decrease) of import, Georgia’s export was further affected by a regional economic downturn in 2014-2016 due to a recession in Russia and falling commodity prices. On the income side of the current account, Georgia experienced a decrease in primary income (including workers’ remittances) where the balance turned negative in 2008. The twofold external shock in 2008 also changed the structure of Georgia’s external financing. Debt financing increased significantly in 2008-2012, as opposed to foreign direct investment (FDI) and portfolio equity inflows. In parallel, the cost of external financing for Georgia increased dramatically, with the implicit interest rate almost doubling by 2008 before gradually decreasing in the following years. The increase in debt financing and the depreciation of the Georgian lari in 2014-2016 resulted in larger external debt. At the same time, Georgia entered the global financial crisis with a relatively low level of foreign exchange reserves. In 2008 and the following few years, international reserves covered neither 3 months of imports nor the short-term external debt. Regarding the fiscal situation, the Georgian authorities responded to external shocks in 2008 with a countercyclical policy. This, together with a drop in revenues, sharply increased the fiscal deficit. A subsequent arrangement with the IMF led to gradual fiscal consolidation. However, Georgia’s fiscal situation deteriorated again in 2014, mainly due to an increase in social spending (for instance, on the Universal Healthcare System introduced in 2013). Importantly, Georgia’s fiscal policy relies on periodic realignment of spending priorities, as revenue flexibility is constrained by the so-called Economic Liberty Act. Georgia’s public debt ratio to GDP increased on the back of the economic slowdown after 2008 and larger fiscal deficits. After a short period of stabilisation, the public debt ratio increased again following the depreciation of the Georgian lari in 2014-2016, as foreign-exchange denominated debt plays a dominant role in Georgia’s public financing. In this difficult macroeconomic context, Georgia benefitted from several arrangements with the IMF. In September 2008, the IMF approved an 18-month Stand-by Arrangement (SBA) of SDR 747 million (USD 1,168 million), aimed at stabilising Georgia’s economy and restoring conditions for strong economic growth. Until the last drawing in July 2010, SDR 577 million (USD 902 million) were disbursed to Georgia. After that, the Georgian authorities treated the arrangement as precautionary and did not draw the instalments that became available, despite successful programme reviews. After the expiry of this first SBA in 2011, Georgia requested another SBA and a Stand-by Credit Facility (SCF). In April 2012, the IMF approved a 24-month SBA and SCF for Georgia with a combined volume of SDR 300 million (USD 464 million). Again, the Georgian authorities treated these arrangements as precautionary and did not borrow from the IMF. After the expiry of the combined SBA and SCF in April 2014, the Georgian authorities requested a further SBA. In July 2014, the IMF approved a 36-month SDR 100 million (USD 154 million) SBA with Georgia to support the authorities’ economic programme. A total of SDR 100 million (USD 123 million) were disbursed until December 2014, after the successful completion of the first review. In the course of 2015, the programme went off-track due to disagreements between the between the IMF and the Georgian authorities on banking supervision, the fiscal strategy and the legal framework for granting of state guarantees.

After these disagreements had been resolved, the IMF approved a 36-month EFF programme of SDR 210 million (USD 285 million) in April 2017. This current arrangement with the IMF supports the reform programme of the Georgian authorities, with a view to promoting higher and more inclusive growth while maintaining macroeconomic stability. To complement Georgia’s arrangements with the IMF, the EU pledged two MFA operations of EUR 46 million each at the International Donors’ Conference in Brussels in October 2008. The first of these operations (EUR 46 million, fully in the form of grants) was implemented in 2009-2010 and is outside the scope of this evaluation. In view of the continuing residual external financing need, Georgia requested the second operation, which is the subject of this evaluation, in May 2010. In January 2011, the Commission proposed an MFA operation of EUR 46 million (half in grants and half in loans). The European Parliament and the Council adopted the MFA decision in August 2013. The MFA operation could not be launched immediately after the adoption due to the fact that Georgia did not have a disbursing, non-precautionary arrangement with the IMF agreement. The approval of a new, disbursing IMF programme in July 2014 allowed the Commission to negotiate with the Georgian authorities a Memorandum of Understanding (MoU) listing specific policy conditions for MFA, a Grant Agreement and a Loan Facility Agreement. All three documents were signed in December 2014. The general objective of this MFA operation, as stipulated in the MFA decision, was to “support economic stabilisation in conjunction with the current IMF programme”. Specific objectives were the “strengthening the efficiency, transparency and accountability, including public finance management systems in Georgia.” In addition, in a Joint Declaration adopted together with the MFA decision, the EU co-legislators stated that MFA “should aim to restore a sustainable external finance situation” as well as “underpin the implementation of a policy programme that contains strong adjustment and structural reform measures designed to improve the balance of payment position […] and reinforce the implementation of relevant agreements and programmes with the Union”. Together with the conditionality of the IMF programmes, the MFA operation aimed to support further structural reforms in Georgia with a focus on reforms in public finance management, the social safety net, the financial sector and the trade and competition policies. The general intervention logic of MFA operations, which is applicable to the 2015-2017 Georgia operation, is summarised in Figure 1 below. Figure 1. Intervention logic of MFA operations Points of comparison The points of comparison, against which the MFA operation of 2015-2017 is assessed, refer to the situation in Georgia (1) prior to the intervention on the one hand and (2) during and immediately after the implementation of the MFA operation on the other hand. As noted in the previous subsection which describes the situation prior to the MFA operation, particular attention is paid to the external sector, the fiscal situation and progress with structural reforms. Georgia’s progress in each of these areas is described in Section 3. 3.Implementation / state of play Economic situation This subsection describes the economic situation in Georgia during and immediately after the implementation of the MFA operation of 2015-2017, with reference to the points of comparison in the external sector and the fiscal situation. The implementation of this operation itself, including Georgia’s progress on structural reforms supported by MFA policy conditionality, is described in the subsequent subsection. As described in detail in chapter 5 of the external evaluation report, Georgia’s balance-of-payments situation gradually improved after the regional slowdown in 2014-2016. The trade deficit decreased starting from 2016, supported by strong export growth. Together with gradual recovery in remittances, the narrowing of the trade deficit helped to decrease the current account deficit. These developments relating to Georgia’s current account and remittances are summarised in, respectively, Figures 2 and 3 below. Figure 2. Structure of the current account (% of GDP)

Figure 3. Remittances (% of GDP) In the financial account, the non-debt financing (FDI and portfolio investment) recovered starting from 2014. At the same, the cost of external financing decreased from 2016. These developments are summarised below in Figures 4 and 5, respectively. Figure 4. External financing (% of GDP) Figure 5. Implicit financing cost of external liabilities (%)

Georgia’s external debt continued to increase, as shown in Figure 6. The main factor behind the increase is the depreciation of the national currency in 2015-2016. Vulnerabilities relating to high level of external debt are to some extent mitigated by the facts that around 80% of the stock is long-term debt and that the cost of financing remains relatively low, reflecting a significant share of concessional financing. On the other hand, around 90% of Georgia’s external debt is denominated in foreign currency, which points to a significant foreign exchange risk. Figure 6. External debt (% of GDP) Georgia’s foreign exchange reserves increased over the reference period (2015-2017) but remained below the level considered to be adequate, as shown in Figure 7. Figure 7. International reserves (USD billion and the IMF ARA metric) In terms of the fiscal situation, the deficit increased slightly during the reference period, as shown in Figure 8. This increase mainly reflects higher government spending (notably, social spending, including healthcare) and was one of the reasons why the IMF SBA of 2014 went off-track. As part of the agreement on a new IMF programme in 2017, the Georgian authorities committed to fiscal consolidation that resulted in subsequent gradual reduction of the deficit. Figure 8. Fiscal deficit (% of GDP) Over the reference period, the Georgian authorities also stabilised and then started to reduce the level of public debt (Figure 9). These public debt dynamics are mainly due to limited public financing needs (which reflect a prudent fiscal policy) and GDP growth (averaging in 3.5% in 2015-2017). Figure 9. Public debt (% of GDP)

Implementation of the MFA operation After the entry into force of the MoU, the Grant Agreement and the Loan Facility Agreement, the EU disbursed to Georgia the first instalment (EUR 13 million in grants and EUR 10 million in loans) of the MFA operation under evaluation. The grant component was disbursed in January 2015 and the loan component in April 2015 (allowing the EU to obtain more favourable financing conditions from the market and pass them on to Georgia). As also required for the first disbursement, Georgia continued to meet the political precondition for MFA (respect for effective democratic mechanisms, rule of law and human rights) and make good progress with the implementation of the IMF programme. For the second instalment (EUR 10 million in grants and EUR 13 million in loans), Georgia was additionally required to implement specific policy conditions set in the MoU that supported structural reforms. The second instalment could not be disbursed as planned in 2015, despite the fact that Georgia implemented these policy conditions (as assessed during a review mission of the Commission staff in November 2015) and continued to meet the political pre-condition for MFA. As explained in Section 2, this was because the IMF SBA of 2014 running off-track. The second instalment was disbursed in May 2017, after the Georgian authorities resolved disagreements with the IMF and agreed on a new programme (EFF) in April 2017. In the area of public finance management, MFA policy conditionality has helped to establish a dedicated training centre, in order to improve the functioning of the electronic public procurement system, as well as to strengthen the operational independence of the State Audit Office. Regarding the social safety net, MFA conditions have helped to improve the management of the Universal Healthcare System and increase the quality of healthcare services by ensuring, respectively, that the Georgian authorities complete a spending survey and establish a quality management unit at the relevant ministry. In the financial sector, MFA conditions have supported the strengthening of capital adequacy in Georgian banks and risk management capacity at the National Bank of Georgia. In terms of trade and competition policies, MFA conditions have supported export performance and strengthened the competition regulation by ensuring, respectively, that the authorities centralise the issuance of certificates of origin and adopt the necessary secondary legislation. The Georgian authorities were effective in the implementation of the conditionality, meeting all of them without the need for waivers. On the EU’s side, the monitoring process was adequate and the Commission complied with all checks ensuring that Georgia had satisfactory fulfilled the reform measures supported by the MFA. Implementation of MFA policy conditions is assessed in detail in Section 5, as part of the relevance and effectiveness of the MFA operation of 2015-2017. Continuing to face external weaknesses, the Georgian authorities requested further MFA from the Commission in June 2017. The proposal of a third MFA in the amount of EUR 45 million (EUR 35 million in loans and EUR 10 million in grants) was adopted by the Commission on 29 September 2017, and approved by co-legislators on 18 April 2018. The Commission has disbursed the first instalment (EUR 15 million loan and EUR 5 million grant) to Georgia in December 2018. The second and final instalment can be disbursed in 2019, provided that the IMF programme remains on track and Georgia meets the agreed policy conditions. 4.Methodology Evaluation techniques used The methodology for evaluating the MFA operation in Georgia of 2015-2017 was guided by the Commission’s Better Regulation Guidelines and the Guidelines for the Ex-Post evaluation of Macro-Financial Assistance Operations. This evaluation was supported by an assessment carried out by an external contractor from September 2018 until April 2019 and was overseen by an Inter-Service Steering Group (ISG). The external evaluation comprised of the following four phases: (1) inception, (2) data collection, (3) analysis, and (4) finalisation. Techniques used for the evaluation are summarised in Figure 10 and described below. These techniques are closely linked to the consultation strategy, which is described in Annex 2 and includes the synopsis of results. Further information on the evaluation techniques is available in Chapter 4 of the external evaluation report. Overall, triangulation of findings obtained using different techniques has helped to increase validity of the evaluation results. Figure 10. Evaluation techniques Evaluation approach – Triangulation Triangulation of findings from different evaluation methods: a.Desk research: -Document analysis; -Quantitative analysis. b.Semi-structured interviews; c.Two focus groups: -Structural and Social Reforms; -Macroeconomic, fiscal and financial sector. d.Delphi survey to key experts. The desk research involved document analysis and quantitative analysis. The main documents used for the evaluation were the MFA Decision, the SWD accompanying the Commission proposal, and implementation reports drafted by Commission’s Directorate-General for Economic and Financial Affairs (DG ECFIN). Additional sources included other EU documents (such as the EU-Georgia Association Agreement), reports of the IMF, World Bank and other international organisations, as well as analysis carried out by research institutes. For quantitative analysis, the evaluation used data from the national sources (mainly, Geostat, the National Bank of Georgia and the Ministry of Finance) as well as from international organisations such as the IMF and the World Bank. The quantitative analysis notably covered macroeconomic fundamentals, fiscal indicators, external sustainability variables, financial sector variables, and structural reforms (e.g. variables measuring institutional development). Additionally, a macroeconomic tool developed by the IMF was used for debt sustainability analysis (DSA), while fiscal savings (resulting from concessional terms of the MFA operation in question, as compared to market-based alternatives) were calculated by comparing the face value and the net present value of the operation. The rest of the techniques – semi-structured interviews, focus groups and a Delphi survey – fall into a category of qualitative analysis.

The semi-structured interviews aimed at gathering information on the design, implementation and results of the MFA operation of 2015-2017, with a focus on the macroeconomic and fiscal situation in Georgia, structural reforms, the social impact and debt sustainability. The interviews were confidential, and pre-interview questionnaires were used to improve the quality. More than 60 stakeholders were interviewed. They included representatives of the Georgian authorities, international financial institutions, as well as donors and external actors (such as experts of economic research institutes). In addition, interviews were carried out with the European Commission (both in the headquarters and in the EU Delegation to Georgia). Meetings of two focus groups were organised in Tbilisi, covering (1) structural and social reforms in Georgia as well as the relevance of MFA policy conditions, and (2) macroeconomic and fiscal developments. Each focus group was formed of eleven participants, but while the first group mainly involved researchers and representatives of non-governmental organisations, the second group targeted representatives of the financial sector. A Delphi survey was undertaken with a panel of 16 experts. Experts were interviewed using a structured questionnaire and were then invited to participate in a second round to reconsider their positions in light of other respondents’ opinions. The results of the Delphi survey, which focussed on the value added of the MFA operation of 2015-2017, are summarised in Annex 2. Risks and limitations While the overall reliability and validity of the evaluation is strong, a number of methodological limitations and challenges effected the evaluation: ·While generally the data coverage is good, it is limited in some areas of interest (notably, as regards social indicators) and longer time series are not always available. An additional limitation relates to fiscal statistics. The Georgian authorities publish several measures of fiscal deficit, without sufficiently explaining the methodological differences. ·Another limitation relates to the timing of the MFA operation in question. The operation was implemented relatively long after its inception, meaning that the inception and the implementation took place under rather different macroeconomic circumstances, which complicates the assessment. ·This changing economic environment over an extended period and the fact that the MFA operation was implemented in parallel with IMF and other international support programmes also makes it difficult to disentangle the impacts of the MFA operation from the impacts resulting from other factors. ·Regarding evaluation techniques, in response to an initial low up-take, the Delphi panel was broadened. This might have resulted in the inclusion of several experts with less detailed knowledge of the MFA operation, as evidenced in a relatively high proportion of “no opinion” answers. Nonetheless, the identified risks and limitations do not put into question the overall reliability of the evaluation, and are mitigated by the fact that results were obtained using different evaluation techniques. Therefore, the conclusions reached in the evaluation can be considered as valid. 5.Analysis and answers to the evaluation questions Evaluation Framework The evaluation covered five evaluation criteria used in the assessment of EU programmes, namely: (1) relevance, (2) effectiveness, (3) efficiency, (4) EU added value, and (5) coherence. The evaluation considered two additional criteria: (6) social impact and (7) debt sustainability.

Answers to questions Evaluation Question 1: Relevance of the operation To what extent were the MFA operation design and outcomes appropriate in relation to the outputs to be produced and the objectives to be achieved? 1.1. To what extent can the MFA design and outcomes be considered to have been appropriate? 1.2. Were the amounts and terms of the financial assistance provided to Georgia adequate?

1.3. Was the conditionality of the MFA operation appropriate in relation to the objectives to be achieved? 1.4. How did the long timeline of the MFA operation impact its relevance? Question 1.1 refers to the relevance of objectives and the overall design of the MFA operation under evaluation. Based on the MFA Decision and the accompanying Joint Declaration and as described in Section 2, the objectives of the MFA operation were, essentially, to alleviate short-term external financing pressure and help Georgia return to a sustainable path. These objectives were relevant in 2011 when the Commission submitted its proposal to the EU co-legislators. As described in Section 3, Georgia was facing external vulnerabilities such as an increasing current account deficit, subdued non-debt financing and high level of foreign currency denominated debt. Stakeholders consulted during this evaluation have confirmed these vulnerabilities. The overall design of the MFA operation was also relevant to its objectives. More specifically, the criteria for the first instalment (ratification of the MoU and the IMF programme being on track) allowed for a swift disbursement. This was in line with the objective to provide short-term relief. The second instalment was additionally subject to the implementation of several policy conditions, which provided a suitable means for encouraging structural reforms. A majority of respondents in the Delphi survey have confirmed that this design (subjecting only the second disbursement to policy conditionality) was the optimal choice. Question 1.2 refers to the amount and terms of the MFA operation of 2015-2017. Georgia’s external financing gap for the period of 2009-2011 was assessed in the Commission SWD accompanying the proposal for the MFA under evaluation to the EU co-legislators. The amount of the EU contribution in the form of two MFA operations, pledged at the International Donors’ Conference in Brussels in 2008, covered 14% of the residual financing gap. The EU contribution, which would have closed the estimated residual financing gap, was considered to be appropriate (in terms of burden sharing with other donors) and proportional (limited to the minimum necessary to achieve short-term macroeconomic stability in Georgia). Despite a long gestation of the MFA operation under evaluation, its amount (EUR 46 million) was still relevant at the time of the actual disbursements. The first instalment was disbursed in January and April 2015 when the external financing needs of Georgia (as reassessed by the Commission and the IMF in 2014) were similar to the level estimated in 2011. As for the second instalment that was disbursed in May 2017, it helped to close the financing gap identified as part of the EFF programme agreed with the IMF staff in April 2017, mostly in form of replenishing Georgia’s foreign exchange reserves. Regarding the terms of the MFA operation of 2015-2017, half of the amount (EUR 23 million) was provided in grants and another half in loans. This particular proportion reflected concerns about the level of Georgia’s public debt (arguing in favour of a grant) and the country’s eligibility for concessional financing from international financial institutions such as the World Bank. At the same time, the EU took into account an improvement in Georgia’s economic situation and the level of development since 2008. Hence, a grant element was maintained but also reduced, compared to the previous MFA operation that was fully in the form of grants. Stakeholders have acknowledged this coherence during the interviews. The results of the Delphi survey have been more diverse: half of the respondents thought that an MFA operation fully in the form of grants would have contributed more to addressing Georgia’s financing needs, while also acknowledging the disciplining effect of loans. Question 1.3 refers to the relevance of MFA policy conditions in relation to the objectives of the operation, as well as to the design of these conditions. As confirmed by the stakeholder interviews and the first focus group on structural reforms, the four areas of MFA policy conditionality – public finance management, the financial sector, the social safety net, and the trade and competition policies – covered the most relevant reform challenges in Georgia. A majority of the respondents in the Delphi survey have also confirmed that all the eight MFA policy conditions were important for structural reforms in the country. Additionally, several conditions (such as strengthening risk management in the banking sector) were highly relevant for the macroeconomic stabilisation objective of the MFA operation under evaluation. According to several stakeholders, some additional measures could have been relevant as MFA policy conditions. Specific examples refer to strengthening the capacity of the Social Services Agency that manages the Universal Healthcare System and supporting responsible lending practices in the banking sector. The Commission acknowledges the relevance of these issues, while also being aware that the list of conditions has to be focused and limited (as also recognised by the stakeholders; see below). In fact, the Commission has taken up some of them as policy conditions for the current MFA operation approved in 2018. Based on the responses in the Delphi survey, the policy conditions of the MFA operation under evaluation were well designed, both in terms of their mix and in terms of their number. The in-built flexibility of some conditions as well as the fact that they were designed to be implemented swiftly after the MoU had been negotiated meant that there was no need for further flexibility during the implementation phase. At the same time, the in-built flexibility of some relevant measures to be implemented swiftly also meant that they had to be defined in less precise terms (e.g. the condition on the establishment of a healthcare quality management unit). The impact of the timeline (from proposal to disbursement) of the operation on its relevance (Question 1.4) was limited. The objectives of macroeconomic stabilisation and supporting structural reforms were just as relevant in 2014 (when the MoU was negotiated and the implementation started) as in 2011 (when the Commission submitted its proposal to the EU co-legislators). In terms of macroeconomic challenges, Georgia was facing a regional crisis that affected both its balance-of-payments and fiscal situation. In terms of structural reforms, the MFA operation under evaluation not only supported them directly during a difficult period but also (indirectly) created space for reforms by easing the macroeconomic adjustment. These views have been confirmed by the Delphi survey and the stakeholder interviews. Evaluation Question 2: Effectiveness To what extent have the objectives of the MFA operation been achieved? 2.1. To what extent has the MFA operation been effective in improving macroeconomic conditions (with focus on the Balance of Payments and exchange rate)? 2.2. To what extent has the MFA operation been effective in terms of fiscal consolidation? 2.3. To what extent have the short and medium-term expected structural effects of the assistance occurred as envisaged?

In order to assess how the MFA operation of 2015-2017 helped to improve Georgia’s balance-of-payments situation (Question 2.1), the evaluation considers the external financing situation and confidence in the economy (as manifested in the market-based financing cost, exchange rates and dollarisation). To quantify the effect of the MFA operation under evaluation on Georgia’s external debt sustainability in the medium (2014-2019) and long (beyond 2019) term, three scenarios have been computed, using the aforementioned IMF’s DSA framework. The first scenario (A) is the baseline and incorporates the (actual) effects of both the MFA operation of 2015-2017 and the IMF SBA approved in 2014. The second (alternative) scenario (B) assumes that neither the MFA nor the IMF SBA were available to Georgia. Finally, the third (alternative) scenario (C) assumes that only the IMF SBA was available. As shown in Figure 11, the combined impact of the MFA and the IMF SBA (baseline, scenario A) helped to limit the increase in Georgia’s external debt. A similar, if more limited, impact is noticeable in the alternative scenario C, assuming that only the IMF operation was available to Georgia. This latter, more limited impact is explained by a smaller size of the MFA operation, compared to the IMF SBA. The aforementioned effects are likewise observable in the long-term, meaning that the MFA operation under evaluation helped to ease external financing pressures in Georgia, as also confirmed by the Delphi survey. At the same time, this long-term trend will depend on Georgia’s ability to address structural issues, e.g. to increase productivity and trade integration. Figure 11. Projections of Georgia’s external debt-to-GDP ratio (%) In terms of confidence in Georgia’s economy, the combined impact of the MFA operation of 2015-2017 and the IMF SBA was also positive. As shown in Figure 12, the external financing cost – measured by the spreads of the Georgian Eurobond over a the US benchmark yield and the JP Morgan Emerging Market Bond Index (EMBI) – decreased in mid-2014 when the IMF SBA was approved and negotiations on the MoU of the MFA operation in question restarted. Figure 12. Spread of a Georgian Eurobond yield over the US benchmark and the EMBI (percentage points)



Similar impacts are also observable in relation to the exchange rates (with the Georgian lari appreciating against the euro and the US dollar in mid-2014) and the dollarisation (the increased confidence being reflected in lower deposit dollarisation). These developments are summarised below in Figure 13 and Figure 14 respectively. Figure 13. Euro and the US dollar exchange rates against the Georgian lari Figure 14. Deposit and loan dollarisation in Georgia (%) It is true, however, that the aforementioned trends relating to the external financing cost, exchange rates and dollarisation have been to some extent reversed in 2015-2016, due to a regional economic slowdown and the accommodative monetary policy that the National Bank of Georgia pursued in response. Question 2.2 refers to the effectiveness of the MFA operation of 2015-2017 in supporting fiscal consolidation in Georgia. The undesignated nature of MFA and the fact that the operation in question was disbursed on highly concessional terms (partly in grants and partly in concessional loans) had a direct, if limited, positive impact on the fiscal position of Georgia. Through this impact, the MFA operation of 2015-2017 helped to smooth the adjustment path and create fiscal space for reforms and sustained social spending. Aside from this short-term impact, some policy conditions attached to the MFA operation had a longer-term positive impact on Georgia’s public finances. This was particularly true of conditions that helped to improve public procurement and the efficiency of healthcare spending. The stakeholder interviews and the Delphi survey have confirmed the view that, while the MFA operation had a limited short-term fiscal impact, its biggest contribution in terms of public finances was through structural reforms. The fiscal consolidation itself took place in the context of programmes agreed with the IMF and complemented by MFA. While the original arrangement (SBA approved in 2014) went off-track in 2015, partly because of the above-target fiscal deficit, Georgia returned to a consolidation path in late 2016. This allowed reaching an agreement on a new IMF programme (EFF approved in 2017), which provided for further gradual consolidation. The short and medium-term structural effects of the MFA operation of 2015-2017 (Question 2.3) largely materialised as planned. The main reason for this was the fact that Georgia satisfactorily implemented all policy conditions attached to the MFA operation, as also confirmed by the stakeholders interviews and the Delphi survey. The latter has also confirmed that MFA conditionality helped to implement the relevant reforms faster than would have been the case without the MFA operation.

More specifically, the public procurement training centre and the relevant certification system were established in 2014. This has helped to improve the functioning of the electronic public procurement system, as also recognised in the Public Expenditure and Financial Accountability Report of 2018. Likewise, the Parliament’s attitude towards the State Audit Office (SAO) has improved, despite the fact that the SAO was occasionally required to provide data that goes beyond its annual financial audit. The Healthcare Management and Utilisation Survey was presented in 2015 and the tasks of a healthcare quality management unit were assigned in 2014. These reforms have helped to improve the efficiency of the Universal Healthcare System, although its financial sustainability remains a concern. In the financial sector, Georgian banks submitted their Internal Capital Adequacy Assessment Process (ICAAP) reports in 2014, while the National Bank of Georgia completed the Supervisory Review and Evaluation Process (SREP) of two largest banks and established a centralised risk management department in, respectively 2015 and 2014. These structural changes have ensured resilience of the Georgian banking sector despite a regional economic downturn of 2014-2016. The issuance of certificates of origin (so-called EUR 1 certificates) has been centralised at the Ministry of Finance, reducing the regulatory burden for exporters. Finally, the secondary legislation relating to the Law on Competition was adopted in 2014, allowing Georgia to establish the Competition Agency that same year. The MFA operation in question also had less direct structural effects, by complementing other international support programmes (as discussed in response to the evaluation question 4 below). Evaluation Question 3: Efficiency of the operation To what extent did the MFA operation design and implementation allow to carry out the intervention efficiently? 3.1. In what way has the design of the MFA assistance conditioned the performance of the operation in respect to its costs and its objectives? 3.2. How did the long timeline of the MFA operation impact its efficiency? 3.3. Was the disbursement of the financial assistance appropriate in the context of the prevailing economic and financial conditions in Georgia? The design of the MFA operation in question ensured efficient performance in relation to costs and objectives (Question 3.1). From the perspective of the EU, efficiency is evident from a relatively low cost of the MFA operation, compared to its significant impact. Whilst the cost to the EU budget was limited to the amount of MFA grants provided, the significant impact came about from the mutual leverage of the MFA operation and other international support programmes such as those of the IMF, which MFA complements . The design of the MFA operation also ensured synergies with other EU policies and instruments, notably with cooperation assistance operations in the areas of public finance management and Georgia’s economic integration with the EU. In this sense, the MFA operation of 2015-2017 was more cost-efficient than would have been the case if various EU Member States had provided the same amount of assistance to Georgia individually. From the perspective of the Georgian authorities, policy measures required as part of the MFA operation under evaluation entailed a direct cost. However, the additional element of this cost was limited: many of these policy measures were part of the national reform agenda anyway and, in some cases, were also required by other international support programmes. In terms of the financial impact, the Georgian authorities would have benefitted more if the whole MFA amount had been provided in the form of grants. At the same time, the highly concessional terms of the loan element (a 15-year maturity, a 10-year grace period and an effective weighted average interest rate of 0.88 per cent) were more attractive than alternative financing options. As already discussed in response to evaluation question 1.4 above, disbursements of MFA were appropriate in the given circumstances, despite the relatively long timeline of the operation (Questions 3.1 and 3.2). At the time of the first disbursement (of a grant and a loan in, respectively January and April 2015), Georgia was facing a difficult balance-of-payments and fiscal situation due to a regional crisis. At the time of the second disbursement, Georgia needed to increase inadequate foreign exchange reserves. The approval of the MFA operation in question took relatively long (more than three years from the official request by the Georgian authorities). This was partly due to procedural disagreements between the European Parliament and the Council regarding a framework regulation for MFA, which put on hold the approval of all new MFA operations. At the same time, the resulting efficiency loss was minimal because of the absence of a disbursing IMF programme (one of the conditions for MFA) from late 2010 until April 2014 and from mid-2015 until April 2017. This would have prevented disbursements of MFA even if the operation under evaluation had been approved earlier. The external evaluators have pointed out that the period from the time when the Georgian authorities submitted the official request for MFA (May 2010) until the time when the Commission submitted its proposal to the EU co-legislators (January 2011) could have been shorter, given that MFA is a rapid response instrument. However, the previous MFA-I operation was only fully disbursed and completed in August 2010, before which the Commission could not start preparing a new proposal. Upon the completion of the MFA-I to Georgia, the Commission prepared the new proposal and adopted it in January 2011. In effect therefore, it took the Commission four months to prepare and adopt the new MFA proposal. The Commission considers this to be within a normal timeframe, yet will endeavour to shorten this period whenever possible, as has already been the case in several subsequent proposals. Evaluation Question 4: EU added-value of the operation What was the rationale for an intervention at EU level? To what extent did the MFA operation add value compared to other interventions by other international donors? 4.1. To what extent have the expected benefits of the EU intervention been attained? 4.2. What is the value resulting from the EU assistance which is additional to the assistance obtained at other levels (IMF, other donors)? 4.3. To what extent has the sharing of roles between the European Commission (DG ECFIN and other DGs), the IMF, Member States and others contributed to optimise the impact of the assistance?

Some direct benefits of the EU intervention (Question 3.1) materialised in the form of implemented specific policy conditions attached to the MFA operation in question (as discussed in response to question 2.3). This has mitigated Georgia’s external vulnerabilities and restored market confidence (as discussed in response to question 2.1). Other, less direct benefits came about from supporting structural reforms in Georgia more generally. Even if many of these reforms were already part of the national agenda, the MFA operation under evaluation helped to prioritise them and generate the necessary political will for their implementation. In addition, the MFA operation has helped to mitigate fiscal pressures (as discussed in response to question 2.2) and resulted in significant savings due to favourable terms of the assistance (as discussed in response to question 7). At the same time, the visibility of the MFA operation was relatively low, with many interviewed stakeholders not being able to distinguish MFA from other means of EU support. The Commission will therefore consider ways of increasing the visibility of the MFA instrument. The additional value of the MFA operation in question, in relation to assistance by other donors, mainly resulted from efficient and complementary sharing of roles between these programmes (Questions 3.1 and 3.2). As an example, the MFA was closely linked to the two IMF programmes (the SBA approved in 2014 and the EFF approved in 2017). MFA disbursements were part of the sources to cover the identified external financing gap. The close link was also evidenced in the deactivation of the MFA operation after the IMF SBA went off-track in 2015 and in a joint push to ensure the independence of the National Bank of Georgia. While there were no cross-conditions (identical policy measures), the Commission and the IMF built their interventions on each other. For instance, the MFA policy condition on capital adequacy in Georgian banks built on the recommendations of the Financial Sector Assessment Programme (FSAP) carried out by the IMF in 2014. A similar sharing of roles was observable in relation to the World Bank. For example, the MFA policy condition on strengthening Georgia’s public procurement system supported the World Bank’s technical assistance in this area. Likewise, the MFA operation in question (notably, policy measures in the area of healthcare) benefitted from the World Bank’s thorough engagement in the public healthcare sector (analytical Public Expenditure Reviews, work on accessibility of healthcare services etc.). The MFA operation also increased the effectiveness of bilateral support that Georgia received from several EU Member States such as Germany and France. As this bilateral support focused on infrastructure development in the energy and transport sectors, it was enhanced by the MFA operation that targeted institutional progress and, in this way, helped to create a more favourable environment for sectorial assistance. Evaluation Question 5: Coherence of the operation To what extent was the MFA operation in line with key principles, objectives and measures taken in other EU external actions towards Georgia? The MFA operation of 2015-2017 was part of a comprehensive package of the EU’s assistance to Georgia. The EU has been providing substantial (regular) cooperation assistance to Georgia, in the form of budget support and technical assistance. In 2007-2013, the EU provided EUR 452 million in assistance to Georgia under the European Neighbourhood Policy Instrument. The indicative allocation of regular cooperation assistance for the period 2014-2020 is EUR 610 million to EUR 746 million under the European Neighbourhood Instrument. Additionally, the European Investment Bank (EIB) supported projects in Georgia with a total loan volume of more than EUR 1.5 billion. As confirmed by stakeholder interviews and the Delphi survey, the EU’s interventions in Georgia were closely coordinated. This coordination involved the relevant Commission services and the EU Delegation to Georgia. The stakeholders have also confirmed that policy conditions of the MFA operation in question reflected the Commission’s long-term agenda. In concrete terms, the MFA operation in question was aligned with the sectorial focus of the EU’s regular cooperation assistance. For instance, in the area of public finance management, the MFA policy condition on strengthening the independence of the State Audit Office was aligned with a cooperation assistance operation aimed at improving external audit. Likewise, a cooperation assistance operation for helping Georgia adapt to the requirements of the Deep and Comprehensive Free Trade Area (DCFTA) with the EU was supported by MFA policy conditions, all of which (except the one relating to risk management in banks) were linked to the requirements the EU-Georgia Association Agreement. Finally, the MFA policy condition on public procurement was aligned with a cooperation assistance operation in the area of public administration reform. Both the EU’s regular cooperation assistance and the EIB lending, which mostly supported infrastructure projects, benefitted from macroeconomic stability and more favourable institutional environment that MFA helped to ensure. In this way, the MFA operation under evaluation increased the effectiveness of other EU support operations in Georgia. Importantly, the MFA operation of 2015-2017 complemented, and not replaced, other forms of the EU’s financial assistance to Georgia. Evaluation Question 6: Social impact of the operation What was the social impact of the MFA operation? 6.1. What were the expected short and medium-term social effects of the assistance? 6.2. To what extent have the expected short and medium-term social effects of the assistance occurred as envisaged? 6.3. What has been the contribution of the MFA to the occurrence of the expected social effects? Given that MFA is primarily about providing an exceptional balance-of-payments support, the operation under evaluation did not directly pursue social objectives. At the same time, the MFA decision refers to beneficiary’s commitment to values shared with the EU such as sustainable development and poverty reduction. Moreover, two policy conditions attached to the MFA operation in question (on the Healthcare Management and Utilisation Survey and the healthcare quality management unit) addressed the social safety net.

In terms of the expected social effects (Question 7.1), the evaluation identified the following expected short-term elements: (1) improvement in the quality and efficiency of the public healthcare services, and (2) smoothing the macroeconomic adjustment path and, in this way, helping to maintain social spending. The MFA operation in question did not specifically target medium-term social objectives but could have been expected to help reduce Georgia’s long-standing social problems, notably high unemployment, poverty and inequality. As explained in Section 4, some limitations due to data availability have been addressed by using other, qualitative evaluation methods. The degree to which these expected social effects materialised is not uniform (Question 7.2). In order to assess changes in healthcare quality (one aspect of the first expected short-term effect), the evaluation considers the neonatal (i.e. newborn) mortality rate, the life expectancy and the share of out-of-pocket expenditure. While the neonatal mortality rate decreased significantly during the reference period (2014-2017), an increase in the life expectancy was more limited, as shown in Figures 15 and 16 respectively. The out-of-pocket expenditure declined both as share of total spending on healthcare and as a percentage of GDP, as shown in Figure 17. This mainly reflects the introduction of the Universal Healthcare System in 2013, which increased public spending and, hence, availability of public healthcare services. Figure 15. Neonatal mortality (rate per 1,000) Figure 16. Life expectancy at birth (years) Figure 17. Out-of-pocket expenditure (% of health expenditure and GDP) At the same time, changes in efficiency of public healthcare services (another aspect of the first expected short-term effect) are less noticeable. According to the interviewed stakeholders, the capacity of the agency that manages the Universal Healthcare System remains limited. This, in turn, limits the agency’s ability to act as an active purchaser of healthcare services from (mostly private) providers. As noted, the Commission acknowledges this issue that is, in fact, being addressed as part of policy conditionality of the current MFA operation approved in 2018. As shown in Figure 18, Georgia maintained an elevated level of social spending (the second expected short-term effect) during the reference period. Spending on healthcare was, in fact, one of the items that increased most. Figure 18. Georgia’s public spending (% of GDP) Assessing the medium-term effects is more complicated because they might have not fully materialised yet, given the fact that the MFA operation was implemented relatively recently. The level of unemployment remained elevated during the reference period due to the skills mismatch and low labour mobility between the regions. A welcome decrease in youth unemployment was largely offset by an increase in unemployment among the older population. The Commission is aware of these issues and is helping to address them through policy conditionality of the current MFA operation. Figure 19. Unemployment (% of active population) As shown in Figure 20, the poverty ratio continued to decrease during the reference period, albeit slower than in earlier years. The aforementioned reduction in the out-of-pocket spending contributed to this decrease. Inequality also decreased but only marginally. Figure 20. Poverty (% of population) and inequality (Gini index, %)

The MFA operation of 2015-2017 contributed to the aforementioned social effects both directly and indirectly (Question 7.3). A direct contribution came in the form of policy conditions that had a positive impact on the quality and efficiency of the public healthcare system, even if a number of problems remain. Indirectly, the MFA operation in question helped to ease fiscal pressures and sustain social spending. Likewise, the MFA operation contributed to restoring macroeconomic stability that supported economic growth and, hence, had a positive impact on employment, income and other social developments. It is difficult to fully disentangle the social effects of the MFA operation under evaluation from other factors (such as support operations of other donors) and quantify them. However, a positive social impact of the MFA operation of 2015-2017 has been confirmed by the stakeholder interviews and the Delphi survey. More specifically, respondents have recognised a significant role that the MFA operation played in maintaining social spending and encouraging reforms in related sectors such as education. Evaluation Question 7: Public Debt Sustainability of the operation What was the impact of the MFA operation on public debt sustainability? 7.1 To what extent has the MFA/IMF assistance contributed to returning the fiscal situation of Georgia to a sustainable path over the medium to longer-term? In order to assess the aforementioned impact, the evaluation considers the sustainability of Georgia’s public debt and fiscal savings resulting from the MFA operation in question. As in the case of projections of external debt, the evaluation uses the IMF DSA framework to quantify the effect on the sustainability of Georgia’s public debt. Three scenarios have been computed for the medium (2014-2019) and long (beyond 2019) term. The first scenario (A) is the baseline and incorporates the (actual) effects of both the MFA operation of 2015-2017 and the IMF SBA approved in 2014. The second (alternative) scenario (B) assumes that neither the MFA nor the IMF SBA were available to Georgia. Finally, the third (alternative) scenario (C) assumes that only the IMF SBA was available. As shown in Figure 21, the combined impact of the MFA and the IMF SBA (baseline, scenario A) helped to limit the increase in Georgia’s public debt. A similar, although more limited, impact is observable in the alternative scenario C, assuming that only the IMF operation was available to Georgia. This latter, more limited impact is explained by a smaller size of the MFA operation, compared to the IMF SBA. The aforementioned effects are likewise observable in the long-term, meaning that the MFA operation of 2015-2017 helped to return Georgia’s public debt to a sustainable path. Analysis shows that these effects materialised mainly through the confidence channel (increased confidence reduced the risk premium on Georgia’s public debt) and the economic growth channel (macroeconomic stabilisation supported growth). Figure 21. Projections of Georgia’s public debt-to-GDP ratio (%) As explained in response to Question 3.1, financial conditions of the MFA operation in question were very favourable. It is therefore appropriate to try to quantify the resulting fiscal savings. This is usually done by comparing concessional financing to regular debt issuances. However, in Georgia’s case, such comparison is not straightforward, as the country has limited market financing of its public debt (a single Eurobond issued in 2011, with a 10-year maturity). In order to estimate the net present value of the MFA grant and loan components in 2015 (at the time of the first disbursement), the evaluation uses two discount rates: (1) 5.25 per cent, which was the average remaining yield to maturity of Georgia’s Eurobond at that time, and (2) 7.35 per cent, which is a sum of the first discount rate and the term premium of 210 basis points. As shown in Figure 22, the resulting fiscal savings of both grant and loan components are estimated at EUR 31.3-33.9 million, or 0.26-0.28% of GDP. As noted, these savings (compared to market financing) helped to smooth Georgia’s adjustment path and create fiscal space for reforms and sustained social spending. Figure 22. Fiscal savings due to MFA (EUR million and % of GDP) Assumed market interest rate Estimated net present value of the loan component in 2015 (in EUR million) Estimated present value of the grant component in 2015 (in EUR million) "Savings" due to the favourable MFA conditions (in EUR million) "Savings" due to favourable MFA conditions (in GDP %) 5.25% 9.3 22.0 31.3 0.26% 7.35% 12.27 21.7 33.9 0.28% 6.Conclusions In August 2013, the European Parliament and the Council approved a Macro Financial Assistance operation of EUR 46 million to Georgia (MFA-II). This was the second of the two operations pledged by the EU at an International Donors’ Conference in Brussels in October 2008, after Georgia’s armed conflict with Russia. Together with the hit from the global financial crisis, the conflict with Russia put the Georgian economy and its external financing under severe pressure, which necessitated international financial support. The aim of this ex-post evaluation of MFA is to assess the relevance, efficiency, effectiveness, coherence, and EU value added of the MFA-II operation to Georgia, which was implemented in 2015-2017. It also explores the social impact of MFA and its effect on Georgia’s public debt sustainability and sets out to draw lessons. Limitations encountered relate predominantly to limited data coverage in some areas (social indicators), the changing economic environment over the extended period during which the MFA-II was implemented, and the fact that MFA-II was implemented in parallel with other EU and international support programmes (making it somewhat difficult to disentangle the impacts of the MFA operation). Nevertheless, these do not put into question the overall reliability of the evaluation analysis as they were mitigated by the fact that information was obtained from a wide range of sources, using different evaluation techniques, alternative scenarios and multiple rounds of feedback. The evaluation finds that the MFA-II was relevant in terms of its objectives, financial envelope and structural conditions. It sought to alleviate short-term external financing pressure, which was present both in 2011 (when the Commission submitted its proposal to the EU co-legislators) and in 2014 (when the MoU was negotiated and the implementation started). The overall design of the operation (subjecting only the second instalment to policy conditionality) is also judged as appropriate. This design allowed a swift first disbursement, to provide short-term relief, while also encouraging structural reforms by means of policy conditionality of the second disbursement.

The areas of MFA policy conditionality covered the most relevant reform challenges in Georgia and were relevant for the macroeconomic stabilisation. The structural effects of the operation largely materialised as planned, as Georgia implemented all relevant policy conditions. Importantly, evidence gathered shows that the relevant reforms were implemented faster than would have been the case without the MFA operation. There is an argument that some additional policy measures in healthcare and banking could also have been relevant, however this has to be balanced with the need to keep the list of MFA conditions focused. Some of these measures have subsequently been included as policy conditions in the current MFA operation to Georgia approved in 2018. The amount of MFA-II was appropriate (in terms of burden sharing with other donors) and proportional (limited to the minimum necessary to achieve short-term macroeconomic stability in Georgia). Moreover, the evaluation finds that the amount was relevant at the time of the actual disbursements, helping to close the financing gap identified as part of Georgia’s programmes with the IMF. The financial terms of the MFA operation (half of the amount was provided in grants and another half in loans) was also appropriate. The evaluation finds that the MFA-II was effective in helping to improve Georgia’s balance-of-payments situation, as well as supporting fiscal consolidation and structural reforms. Indeed, together with the IMF arrangements, the MFA operation helped to limit the increase in Georgia’s external debt and increase confidence in the Georgian economy (by decreasing the market-based financing cost and dollarisation, as well as by helping to stabilise the national currency). The MFA-II was designed and implemented efficiently, and in close coordination with the Georgian’s authorities, the IMF and the World Bank. Delay in the adoption was caused by procedural disagreements between the co-legislators. However, the evaluation finds that this exceptionally long timeline of approving the operation did not result in efficiency losses, because from late 2010 until July 2014 and from mid-2015 until April 2017, Georgia did not have a disbursing IMF programme (a precondition for granting MFA). The latter would have prevented disbursements of MFA even if the operation under evaluation had been approved earlier. Moreover, the Commission asserts that the timeframe between when the Georgian authorities submitted the official request for MFA request (May 2010) until the time when the proposal was submitted to the EU co-legislators (January 2011) is in effect within normality, as the Commission could only start preparing a new proposal once the previous MFA-I operation was fully disbursed and complete (August 2010). At the same time, the Commission will endeavour to shorten this period whenever possible, as has already been the case in several subsequent proposals. In the subsequent MFA programme to Georgia for instance, the Commission adopted the proposal for further MFA roughly three months after it received the request from the Georgian authorities. The issue of MFA timelines will be further examined in a wider MFA meta-evaluation (elaborated in Annex 1) planned to commence in 2020. The MFA operation under evaluation was a coherent part of a comprehensive package of the EU’s assistance to Georgia. Indeed, the evaluation confirms close coordination between MFA-II and other EU's interventions. The EU’s added value was most apparent in stimulating the structural reform process in Georgia. In addition, the MFA operation helped to mitigate fiscal pressures and resulted in significant savings due to favourable terms of the assistance. Despite these benefits, the evaluation finds that the visibility of MFA-II was relatively low. The issue of MFA operational visibility will also be further examined in a MFA meta-evaluation planned to commence in 2020. The MFA-II had an overall positive social impact. It helped to improve the quality and efficiency of the public healthcare services, although progress in this area is uneven and is being addressed as part of policy conditionality of the current MFA operation approved in 2018. The MFA operation also helped to smooth the macroeconomic adjustment path and, in this way, maintain social spending. More broadly, the evaluation asserts that the MFA operation contributed to restoring macroeconomic stability that supported economic growth and, hence, had a positive impact on employment, income and other social developments. Finally, the evaluation finds that the MFA-II had a positive effect on the sustainability of Georgia’s public debt and enabled fiscal savings. Together with the IMF arrangements, the MFA-II helped to limit the increase in Georgia’s public debt. The fiscal savings of around 0.3% GDP resulted from very favourable financial conditions of the MFA operation. These savings (compared to market financing) helped to smooth Georgia’s adjustment path and create fiscal space for reforms and sustained social spending. Annex 1: Procedural information Organisation, design and timing This evaluation assesses the MFA operation in Georgia, in the amount of EUR 46 million, which was approved in 2013 and implemented in 2015-2017. The assessment is in line with the Financial Regulation (Article 30(4)) and the relevant MFA Decision, which require the European Commission to submit an ex-post evaluation report to the European Parliament and the Council. The objective of the evaluation is to draw lessons with respect to the EU’s financial assistance, in particular the design and implementation of the programmes and the way they contributed to achieving macroeconomic stabilisation and fostering structural reforms. Apart from identifying areas of improvement for similar on-going or future possible interventions, the evaluation also aims at ensuring better transparency and accountability of the Commission’s activities. The evaluation looks at various aspects of this particular EU intervention (relevance, effectiveness, efficiency, EU added-value, coherence with other EU policies towards Georgia, as well as the social impact and the impact on the sustainability of Georgia’s public debt). In order to ensure validity, the analysis and conclusions are based on the evidence obtained using several evaluation methods (desk research, stakeholder interviews, focus groups and a Delphi survey). DG ECFIN, which was the lead DG, chaired the ISG that was set up to manage the evaluation. Apart from DG ECFIN, the ISG comprised of representatives of other Commission services (the Secretariat General and the Directorate-General for Neighbourhood and Enlargement Negotiations) and the EEAS. DG ECFIN hired an external contractor to complete an evaluation report, which informs this SWD. A kick-off meeting, where the ISG and the external contractor discussed the deliverables and the evaluation methods, took place in September 2018. This was followed by meetings on the inception and interim reports in, respectively, October and December 2018. The final report was submitted in April 2018. In addition to meetings, ISG members were continuously informed and consulted (via email and by phone) during the evaluation. The work of the external contractor was complemented by internal analysis.

Consultation of stakeholders The objective of stakeholder consultation was to collect as much valuable and relevant economics-based inputs as possible from various groups and people involved to construct an ex-post assessment of the design, implementation and impact of the MFA-II operation. This did not include an open public consultation, but was rather targeted towards specialists - people with an informed economic understanding of the Georgian MFA-II operation and the context in which it was implemented either by being closely involved its development and/or the implementation or persons with expert knowledge in the areas related to the objectives of the operation. In addition, an indicative evaluation roadmap was published in June 2018 to seek wider feedback. Finally, a number of the conclusions and limitations identified will require further consideration and may benefit from cross-comparison with other recently completed MFA operations. A meta-evaluation is planned for 2020 and it will explore these points in more detail. The meta-evaluation will provide a synthesis of the results of the evaluation studies of MFA operations carried out in the last ten years and will consider the reliability and relevance of the methodology, process, outputs, and outcomes of these evaluations. In addition the exercise will assess the timeline of MFA operations and potential linkages to added-value and operational visibility. An open public consultation will be undertaken as part of the meta-evaluation to gather the views of wider stakeholders.



Annex 2: Stakeholder consultation strategy, including summary of results

This stakeholder consultation strategy was developed with the overall objective to capture as much valuable and relevant information as possible with regard to the MFA-II operation in addition to information collected through review of key documentation and communication, consultation of EC officials, and data analysis. The consultation focused on extracting recollections from the time period in which the operation was designed (2011–2014) and implemented (2015–April 2017), but also on collecting views on the period after the MFA-II was ended (May 2017– 2018) to assess its impact and sustainability. This consultation strategy: ·sets out the objectives of the consultation; ·maps key stakeholders; ·presents the consultation methods and tools which are used; and ·demonstrates how the stakeholder consultation fits in the evaluation framework. Table A.1.1 presents a detailed timeframe for the implementation of this consultation. The items listed in this timeframe are elaborated in the following sections. CONSULTATION TIME SCHEDULE OCTOBER 2018 NOVEMBER 2018 JANUARY 2019 Brussels 9; 17/19 October Home based 20 November Tbilisi, Georgia 12-16 November Home-based 22-27 November Tbilisi, Georgia 15-18 January Home-based Week of January 21 Georgia authorities Semi-structured interviews Georgia authorities Semi-structured interviews IFIs, Other donors IFIs Delphi survey (online survey + follow up) Other donors IFIs Delphi survey (online survey + follow up) EC representatives Semi-structured interviews IMF Semi-structured interviews External experts Focus group structural and social reforms Georgia authorities External experts Focus group macroeconomic and fiscal reforms Georgia authorities Objective setting The objective of stakeholder consultation was to collect as much valuable and relevant information as possible from various groups and people involved to construct an ex-post assessment of the design, implementation and impact of the MFA-II operation. The external evaluators consulted stakeholders on the following key aspects: ·Relevance of the MFA-II operation: the relevance of the objectives, the financial envelop and the conditionality, both at the time of designing the MFA-II operation (2011-2014) and with the benefit of hindsight; ·Effectiveness of the MFA-II operation: the direct results of the operation, i.e. its results on macroeconomic level and in the area of the structural reforms;

·Efficiency of the MFA-II operation: the design and process of the MFA-II in terms of value-for-money; ·EU-added value of the MFA-II operation: the added value of the operation when considering other possible scenarios and alternatives; ·Coherence of the MFA-II operation: alignment with other support initiatives implemented at the time of the MFA-II; ·Social impact: the more indirect impact of the MFA-II operation in the context of social development in Georgia; ·Debt sustainability: the longer-term result of the MFA-II in terms of implications to Georgia’s government and external debt dynamics and the fiscal and external sustainability. Consultation was thus partly related to recalling past events, but also to collect current opinions, which can be made with the benefit of hindsight. The aim was thus to gain an understanding of the decision-making at the time of the design and implementation of the MFA operation, but also to identify the actual relevance and impact of the operation. While stakeholders were asked to make (subjective) assessments and express their personal opinions, they were encouraged to refer to written sources wherever possible. Eventually, the results of the stakeholder consultation were triangulated with data and documentation to provide well-evidenced responses to Evaluation Questions (as demonstrated in the last section of this annex). Stakeholder mapping Consultation was targeted to specialists – either people who have either been closely involved in the development and/or the implementation of the MFA operation or people with expert knowledge in the areas related to the objectives of the MFA operation (i.e. macroeconomic and fiscal policy, and structural reforms in the areas of PFM, social policy, financial sector, trade and competition policy). Below are the four groups of stakeholders that had a central role in this consultation strategy: 1.Georgia public Institutions Obviously, the recipient was an important stakeholder to consult on the key aspects, in order to incorporate the beneficiary’s view on the MFA2 operation. After having gone through all documentation provided and collected on the MFA-II, the following key institutions within the Government of Georgia were invited for a discussion on the design and the implementation MFA-II operation, and its macroeconomic and fiscal effects: ·Ministry of Finance (MOF): implementing Ministry of the MFA loan;

·National Bank of Georgia (NBG): implementing financial institution; ·Parliamentary Budget Office (PBO). Within these organisations, a distinction should be made between high-level policy makers and the technical staff working on actual implementation. The latter group is able to specifically comment on the efficiency of the implementation of the MFA-II operation. Furthermore, a number of other key stakeholders within the government were consulted, specifically on the relevance, effectiveness, and impact of the conditions for structural reforms: ·State Audit Office (SAO): on PFM reforms (action 1 and 2); ·State Procurement Agency on action 1; ·Ministry of Labour, Health and Social Affairs on social reforms (action 3 and 4); ·National Bank of Georgia (NBG) on financial sector reforms (action 5 and 6); ·Ministry of Economy and Sustainable Development on trade and competition policy (action 7 and 8). 2.International Financial Institutions As major international financial institutions (IFIs), the IMF and World Bank were key stakeholders for consultation as they were to some extent involved in the preparation and implementation of the MFA-II. In addition, they provided similar support to Georgia (the IMF via a SBA and an EFF, the World Bank via DPLs). Both organisations could thus provide input to all key aspects of the evaluation. They are probably not the group with the largest interest in this MFA evaluation, but they do have a significant influence. 3.Georgia external (i.e. non-governmental) experts There is a variety of actors, who were not directly involved in the MFA operation itself, but are very knowledgeable on the topic of macroeconomic and fiscal developments, and on structural and social reforms in Georgia. It was important to consult these actors as well in order to determine the actual relevance, effectiveness and impact of the MFA-II, as they possess the knowledge to place the MFA-II in the wider context of Georgia’s economic and social situation. These actors include: ·Ex-government officials and (ex-) Parliamentarians: To give, wherever possible, a more external view on the structural conditions of MFA; ·NGO, academics and other interest groups: To provide an outsider’s view on the economic and social developments that have taken place in Georgia in the period of 2011-2018 and on the structural reforms which have taken place in Georgia since the MFA; ·Banks and financial institutions: Officials from private banks in Georgia could provide an external view on the economic and financial developments in Georgia and on the current macroeconomic and fiscal situation.

4.Other donors A selection of other donors was consulted, to provide a more outsider’s’ opinion on the MFA operation in the context of wider aid provisions and to gain further insight in the coherence of the MFA operation. They do not have a strong interest or influence, but are interesting to get a better understanding of the context in which the MFA was provided, and possibly can put the relevance and impact into context. These include: ·US Agency for International Development; ·German Development Agency, GIZ; ·World Health Organisation. Two focus group sessions were organized with a distinct focus. The first session covered structural and social reforms in Georgia, and focused on the relevance of the MFA conditions. For this focus group, the invitees consisted of (former) Parliamentarians, academics and non-governmental organisations. Representatives of the following organisations participated in the first focus group discussion: Ilia State University Tbilisi State University Transparency International - Georgia The Association of Young Financiers and Businessmen; The Georgian Economics Parliamentary Committee on Budget and Finance Georgian Taxpayers Union ISET PMC Research European Bank for Reconstruction and Development.

The second focus group session focused on the macroeconomic and fiscal developments, including topics like Georgia’s financing needs, debt sustainability as well as the financial sector reforms. Participants were experts from the financial sector. The focus was on senior economists residing in special economic research or analysis departments of the banks and other financial institutions. (See the detailed list of the participants below). Representatives of the following organisations participated in the first focus group discussion: Procredit Bank Georgia Investment Arm of Bank of Georgia Georgia Capital PLC Basis Bank FINCA Bank - Georgia, VTB Bank MFO Crystal Terabank Pasha Bank TBC Capital. The full list of participants is detailed in Annex 1 of the external report. Consultation methods Related to the four groups of key stakeholders above, the evaluators used a targeted consultation approach, utilising three key tools: 1.Semi-structured interviews The objective of the interviews was to extract detailed information on the following:

·MFA design and implementation; ·results of MFA on the macroeconomic and fiscal situation; ·results in the fields of the structural reform conditions; ·social impact; ·debt sustainability. Interviews were thus carried out particularly with the first two stakeholders, who are well aware of the MFA instrument and its implementation. Interviews were also conducted with the fourth group (other donors), but these interviews were focused on the coherence of MFA with other donor initiatives and were less in-depth. The format of semi-structured interviews was chosen on purpose: on the one hand, this format offers the possibility to discuss a few set topics with the interviewees. Details were asked on events which happened in the past, therefore pre-interview questionnaires were sent out. These questionnaires contained a brief overview of key bullet points that the evaluators would like to discuss, to enable the interviewee to prepare him/herself by collecting information in advance. On the other hand, semi-structured interviews leave room for the interviewer to raise other relevant issues, also in feedback to answers of the interviewee. The most important findings of the semi-structured interviews are listed in the following table:

No. Most important findings of the semi-structured interviews 1. The blend of grant and medium-term concessional loan of the assistance was appropriate. 2. The most important reform areas were covered by the conditions and the actions set by the MoU were all relevant or highly relevant. 3. Additional measures could have targeted (i) ensuring the conditions for SAO’s revenue auditing; (ii) improvement of internal auditing; (iii) improving the human and infrastructural capacities of Social Services Agency (iv) supporting responsible lending practices; (v) support filling the knowledge gap of companies related to the DFCTA; and (vi) improving the human and infrastructural capacities of the Competition Agency. 4. Public procurement needs to be further developed. Lack of sufficient progress in the area of the health care (cost effectiveness) and the competition policy was mainly due to the lack of sufficient human and infrastructural capacities. Implementation of the DFCTA is very costly and the beneficial impact on trade performance is not yet tangible. 5. The primary added value of the operation is its important role in promoting structural reforms. Getting an external credibility stamp on policy reforms is a highly important factor in Georgia. 6. MFA-II also had a prominent role in helping to alleviate external pressure, primarily by supporting the replenishment of FX reserves and in helping to restore market confidence. 7. The short-term impact on budget financing was a less important contribution of the MFA operation. 8. The conditions were coherent with and well complemented the operations of other IFIs, which was well appreciated by non-IFI stakeholders as well. There was an efficient division of labour among IFIs. 9. The pace and the ambition of the fiscal consolidation plan set by the IMF was reasonable. Deficit targets were missed due to the lack of the government’s commitment. 2.Expert focus group Focus groups aim to gain information of the MFA in a wider context: what has been its relevance and its impact if the operation is put in a broader perspective. Focus groups are ideal for exploring people's experiences, opinions, wishes and concerns and have been identified as especially useful for studying the success or failure of particular policies and programmes. Specifically for this MFA, organising these focus groups helped in understanding the current paradigm of reforms and get a better understanding on how MFA has been tailored to the local situation in Georgia. This instrument was particularly useful for the third stakeholder group: these experts have been or are too far away from the MFA operation to conduct detailed one-on-one interviews, but their participation in a group discussion would be very useful to gain deeper understanding of the macro/fiscal developments and the structural/social reforms in Georgia. Discussion were restricted to two hours, to encourage participation of people and ensured participants that Chatham House rules were to be applied. These rules elicit the maximum amount of input from the participants and therefore provide the best opportunity to contribute to addressing the evaluation questions with valuable stakeholder insights that may not be possible in a more open forum. Two group sessions were organised, each with a specific focus:

a.Structural and social reforms in Georgia ·Relevance of the MFA conditions in light of Georgia’s developments; ·Reforms in: PFM, social safety net, trade and competition policy; ·Development of social indicators regarding employment and poverty and inequality. b.Macroeconomic, fiscal and financial sector developments in Georgia ·Georgia’s financing requirements; ·Internal and external factors of Georgia’s external financial situation; ·Pace, ambition and composition of appropriate financial consolidation; ·Debt sustainability; ·Financial sector reforms. The first focus group took place during the first mission to Georgia in November 2018. For this focus group, invitees included (former) Parliamentarians, academics and non-governmental organisations (see list of participants above). The second focus group took place during the second mission in January 2019 and involved officials from banks and financial institutions (see list of participants above). The most important findings of the Focus Group discussions are listed in the following table: No. Most important findings of the Focus Group discussions First Focus Group on structural and social reforms 1. Most of the important reform areas were covered by the conditions. 2. Conditions et in the MoU were relevant. 3. One of the most relevant action was the one supporting the independence of SAO. Independence of SAO strengthened and the office produces great reports, but the recommendations are not always implemented due to the lack of political will. 4. Low cost effectiveness of the UHC is a major problem of the health care system. 5. Level of social expenditure is high, there is a need to improve its efficiency. 6. High unemployment and skill mismatches are lingering problems of the Georgian economy. Second Focus Group on macroeconomic, fiscal and financial sector developments 1. The IMF/EU programme caused a 110 basis points decrease in the country risk premium. 2. Group discussions gave important inputs for the long term assumptions to the debts sustainability analysis. 3. ICAAP condition (Action 5) was highly relevant both in terms of the structural reform process and the economic challenges. 4. Additional measures could have support the introduction of responsible lending practices. Lack of appropriate regulation in the past and over indebtedness of low income households is a pressing problem for the economy. 5. External debt sustainability, high dollarization (both in the public and private balance sheets) are still relevant problems of the economy. 3.Delphi method The Delphi method is an evaluation methodology that relies on judgmental estimates of experts based on their insights and collective knowledge. The external evaluators applied a light version of the Delphi method as an additional consultation tool. The objective of using this tool was to gain further insight into the added value of the MFA operation. In November, after the first field mission, the evaluators identified a few possible scenarios related to the Georgia’s macroeconomic and fiscal developments and structural reforms. The main question to the participants in the Delphi survey was to what extent the MFA operation has contributed to certain macroeconomic and fiscal developments, and in the field of structural reform, by considering what would have happened if the MFA would not have been granted.

In operational terms, the Delphi method involved two rounds of consultations. The external evaluators sent the panel members a link to an electronic questionnaire powered by the CheckMarket Survey Tool. They also provide them the opportunity to re-assess their position. The panel included 16 experts. It is a balanced mix of Georgian officials, EU officials, either based in Brussels or in-country, and representatives of other IFIs. This allowed us to receive a sufficient amount of valid responses. It should be emphasised that the quality of result is not so much related to the number of respondents, but rather to the level of knowledge of the expert panel. Regarding the survey instrument, the experience with previous MFA evaluations strongly suggested the use of a simplified questionnaire, focusing on a limited set of key variables. This increases the response rate and favour the emergence of a consensus opinion. The most important findings of the Delphi survey are listed in the following table: No. Most important findings of the Delphi survey 1. The impact of MFA has been primarily through the promotion of structural reforms and the easing the balance-of-payment pressure in Georgia. 2. The general design of the operation (i.e. with the first instalment linked to the IMF SBA progress in general and the second instalment linked to the structural conditions) was optimal. 3. The design of the conditionality was appropriate. The MoU included a balanced mix of conditions, the complexity and the number of the actions were proper. 4. The conditions set by the MoU were important or fairly important. 5. All conditions were completed satisfactorily. 6. The main value added of the conditionality was through the speeding up the implementation of the reforms. 7. The alignment of the conditions with other support programmes was appropriate. 8. The MFA operation made a significant difference in enabling Georgia to sustain the costs for social provisions. The non-social related structural conditions had a sizeable indirect positive impact on social developments. 9. The social safety net conditions attached to the MFA helped to reinforce or kick-start some reforms in the social sector. The consultation strategy and the evaluation framework Table A.1.4 combines the different groups of stakeholders and the consultation methods, and shows how they contributed to answering the evaluation questions. The table also mentions other sources, which provided information that could be triangulated with the information collected from stakeholder consultation. EQ criteria Sub-criteria Stakeholders Consultation method Other sources RELEVANCE Relevance of objectives Georgia Authorities: MOF, NBG Semi-structured interviews ·Consultation of EC staff; ·Study of documentation. IFIs · Other donors · Georgia authorities, IFIs Delphi survey · External experts (officials from banks and financial institutions) Focus group on macroeconomic and fiscal/financial topics · Relevance of the financial envelope Georgia Authorities: MOF, NBG Semi-structured interviews Delphi survey ·Consultation of EC staff; ·Economic/financial background papers. IFIs · External experts (officials from banks and financial institutions) Focus group on macroeconomic and fiscal/financial topics · Relevance of conditions Georgia Authorities: selected organisations/ ministries for structural reforms Semi-structured interviews Delphi survey ·Consultation of EC staff; ·Case studies on financial sector and trade policy; ·Reports on socio-economic situation in Georgia. IFIs - External experts (Parliamentarians and NGO/academics) Focus group on structural reforms - EFFECTIVENESS Effectiveness in terms of macroeconomic stabilisation Georgia Authorities: MOF, NBG Semi-structured interviews ·Consultation of EC staff; ·Debt sustainability analysis; ·Data analysis; ·Document review of macroeconomic reports. IFIs · · External experts (officials from banks and financial institutions) · External experts (officials from banks and financial institutions) Focus group on macroeconomic and fiscal/financial topics · Effectiveness in terms of fiscal policy Georgia Authorities: MOF, NBG Semi-structured interviews ·Consultation of EC staff; ·Descriptive quantitative analysis; ·Document review of fiscal reports. IFIs · External experts (officials from banks and financial institutions) Focus group on macroeconomic and fiscal/financial topics · Effectiveness in terms of structural reforms Georgia Authorities: selected organisations/ ministries for structural reforms Semi-structured interviews ·Consultation of EC staff; ·Data analysis; ·Document review of Georgia reforms; ·Case studies on financial sector and trade policy. IFIs Semi-structured interviews · External experts (Parliamentarians and NGO/academics) Focus group on structural reforms · EFFICIENCY NA Georgia Authorities: MOF, NBG (more technical staff) Semi-structured interviews ·Consultation of EC staff; ·Desk research, review of documentation/communication. EU ADDED VALUE NA Georgia Authorities: MOF, NBG Semi-structured interviews ·Delphi survey ·Delphi survey to key EC staff; ·Documentation review and interviews with EC staff on the rationale. IFIs · COHERENCE NA Georgia Authorities: MOF, NBG Semi-structured interviews ·Consultation of EC staff; ·Review of programme documentation and EU / other donor programmes; ·Case studies. Other donors Semi-structured interviews · SOCIAL IMPACT NA Georgia Authorities: selected organisations/ ministries for structural reforms Semi-structured interviews; Delphi survey ·Consultation of EC staff; ·Document and data analysis of social variables. IFIs · Other donors · External experts (Parliamentarians and NGO/academics) Focus group on structural reforms · DEBT SUSTAINABILITY NA Georgia Authorities: MOF, NBG Semi-structured interviews ·Consultation of EC staff; ·Debt sustainability analysis; ·Document review of fiscal reports. IFIs · External experts (officials from banks and financial institutions) Focus group on macroeconomic and fiscal topics · List of key MFA-II documents reviewed European Commission DG ECFIN (2010). Proposal for Further Macro-Financial Assistance to Georgia for 2010-11, Note for the Economic and Financial Committee. ECFIN/D2/AJ/ic Ares(2010)SN680365. European Commission DG ECFIN (2011a). Proposal for a Decision of the European Parliament and of the Council providing further macro-financial assistance to Georgia. 13.1.2011, COM(2010) 804 final. European Commission DG ECFIN (2011b). Ex-ante evaluation statement on further macro-financial assistance to Georgia, Accompanying document to the Proposal for a Decision of the European Parliament and of the Council providing further macro-financial assistance to Georgia. 13.1.2011, SEC(2010) 1617 final.