COMMISSION STAFF WORKING DOCUMENT EVALUATION of Decision No 778/2013/EU of the European Parliament and of the Council of 12 August 2013 providing further macro-financial assistance to Georgia
Date: 2019-12-17
Institutions: Directorate-General for Economic and Financial Affairs, European Commission
Subjects: EU aid, EU loan, European neighbourhood policy, Georgia, economic reform, economic stabilisation, financial aid, macro-financial assistance

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 17.12.2019  SWD(2019) 447 final  COMMISSION STAFF WORKING DOCUMENT  EVALUATION of Decision No 778/2013/EU of the European Parliament and of the Council of 12 August 2013 providing further macro-financial assistance to Georgia {SWD(2019) 449 final}  
 
 
Table of contents 

GLOSSARY ........................................................................................................................ 2 

1. 

INTRODUCTION ....................................................................................................... 3 

Purpose of the evaluation 
Scope of the evaluation 

3 
3 

2.  BACKGROUND TO THE INTERVENTION ........................................................... 4 

Description of the intervention and its objectives 
Points of comparison 

4 
7 

3. 

IMPLEMENTATION / STATE OF PLAY ................................................................ 7 

Economic situation 
Implementation of the MFA operation 

7 
11 

4.  METHODOLOGY .................................................................................................... 12 

Evaluation techniques used 
Risks and limitations 

12 
14 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS ................ 15 

Evaluation Question 1: Relevance of the operation 
Evaluation Question 2: Effectiveness 
Evaluation Question 3: Efficiency of the operation 
Evaluation Question 4: EU added-value of the operation 
Evaluation Question 5: Coherence of the operation 
Evaluation Question 6: Social impact of the operation 
Evaluation Question 7: Public Debt Sustainability of the operation 

15 
17 
21 
22 
23 
24 
28 

6.  CONCLUSIONS ....................................................................................................... 29 

ANNEX 1: PROCEDURAL INFORMATION ................................................................ 33 

Organisation, design and timing 
Exceptions to the Better Regulation Guidelines 

33 
33 

ANNEX  2:  STAKEHOLDER  CONSULTATION,  INCLUDING  SUMMARY 

OF RESULTS ............................................................................................................ 34 

ANNEX 3: TIMELINE OF THE MFA OPERATION ..................................................... 49 

1 

 
 
Glossary 

Term  

Meaning or definition 

DCFTA  
DG ECFIN 
DSA 
EC 
EFF 
EIB 
EMBI 
EQ 
EU 
EUR 
FDI 
FSAP 
FX 
GDP 
GEL 
ICAAP   
IFI 
IMF 
ISG 
LHS 
MFA 
MOF 
MoU 
NBG 
NGO 
PFM 
PPP 
RHS 
SAO 
SBA 
SCF 
SREP 
SWD 
ToR 
UHC 
US 
USD 

Deep and Comprehensive Free Trade Area 
Directorate-General for Economic and Financial Affairs 
Debt Sustainability Analysis 
European Commission 
Extended Fund Facility 
European Investment Bank 
JPMorgan Emerging Market Bond Index 
Evaluation question 
European Union 
Euro 
Foreign Direct Investment 
Financial Sector Assessment Programme 
Foreign exchange 
Gross Domestic Product 
Georgian lari 
Internal Capital Adequacy Assessment Process 
International Financial Institution 
International Monetary Fund 
Inter-Service Steering Group 
Left hand scale 
Macro-financial assistance 
Ministry of Finance 
Memorandum of Understanding 
National Bank of Georgia 
Non-governmental organisation 
Public Finance Management 
Public Private Partnership 
Right Hand Scale 
State Audit Office 
Stand-By Agreement 
Stand-By Credit Facility  
Supervisory Review and Evaluation Process 
Staff Working Document 
Terms of Reference 
Universal Healthcare 
United States 
US dollar 

2 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1. 

INTRODUCTION 

Purpose of the evaluation 

This  Staff  Working  Document  presents  an  evaluation  of  Macro-Financial  Assistance 
(MFA)  to  Georgia,  which  was  implemented  in  2015-2017  (MFA  operation  of  2015-
2017).  The  evaluation  is  based  on  a  study  undertaken  by  an  external  contractor  and  a 
consultation  process  that  involved  various  stakeholders,  as  well  as  European  Union 
(EU) staff working with Georgia.  

The aim of the evaluation is to assess whether the MFA operation of 2015-2017 met its 
objectives  to  support  Georgia  in  addressing  its  balance-of-payments  problems  and 
implementing  economic  and  structural  reforms  that  would  stabilise  its  economy  and 
enhance  the  sustainability  of  its  external  position.  The  purpose  of  the  evaluation  is  to 
inform  the  future  use  of  the  MFA  instrument,  while  also  ensuring  transparency  and 
accountability. 

Scope of the evaluation 
The  MFA  instrument  is  a  form  of  undesignated1  financial  aid  extended  by  the  EU  to 
partner  countries  experiencing  a  balance-of-payments  crisis.  MFA  takes  the  form  of 
medium/long-term  loans,  grants  or  a  combination  of  the  two.  It  is  designed  for  third 
countries that are geographically, economically and politically close to the EU. Unlike 
other, regular development aid provided by the EU, MFA is exceptional in nature and is 
mobilised  on  a  case-by-case  basis.  Its  objective  is  to  restore  a  sustainable  external 
financial  situation,  while  encouraging  economic  adjustment  and  structural  reforms. 
MFA generally complements financing provided in the context of a reform programme 
agreed with the International Monetary Fund (IMF). 
In  August  2013,  the  European  Parliament  and  the  Council  adopted  a  decision2  to 
provide  EUR  46 million  of  MFA  to  Georgia.  This  was  the  second  MFA  operation  to 
Georgia  since  the  country’s  military  conflict  with  Russia  in  August  2008.  In  October 
2008, the EU pledged two MFA operations of EUR 46 million each at the International 
Donors’ Conference in Brussels. The first of these operations was implemented in 2009-
2010  and  was  evaluated  in  2011-20123.  The  second  of  these  operations  was 
implemented  in  2015  and  2017,  and  is  the  subject  of  this  evaluation.  The  timeline  in 
Annex 3 lists the key milestones of the operation. 

In  accordance  with  the  Financial  Regulation  (Article  30(4)),  MFA  operations  in  third 
countries  are  subject  to  an  ex-post  evaluation.  In  turn,  the  aforementioned  MFA 
decision for Georgia stipulates that the European Commission is required to “submit to 
the European Parliament and to the Council an ex post evaluation report.” 

To this end, DG ECFIN hired an external contractor to complete an evaluation report, 
which informs this Staff Working Document (SWD). The external evaluation ran from 
September 2018 until April 2019.  

1   In terms of the use of funds. 
2   Decision No 778/2013/EU (OJ L 218, 14.8.2013, p. 15–23). 
3   Ex-Post Evaluation of MFA-I Georgia available at: 

https://ec.europa.eu/dgs/economy_finance/evaluation/pdf/mfa_georgia_final_report-apr_en.pdf 

3 

 
 
 
                                                           
 
The objective of this ex-post evaluation is twofold:  

1.  To analyse the impact of the MFA on the economy of Georgia, and in particular, 

on the sustainability of its external position; and  

2.  To  assess  the  added  value  of  the  EU’s  intervention.  In  general,  the  evaluation 

seeks to draw lessons with respect to the EU’s financial assistance, i.e. 

a.  Whether the ex-ante considerations determining the design and terms of 
the  operation  were  appropriate,  taking  due  account  of  the  economic, 
political and institutional context; and 

b.  Whether the outcome of the programme met the objectives. 

These  main  areas  are  assessed  along  the  following  key  evaluation  criteria:  relevance, 
effectiveness,  efficiency,  EU  added-value,  and  coherence  with  other  EU  policies.  In 
addition,  the  evaluation  assesses  the  social  impact  of  the  MFA  and  the  impact  on  the 
sustainability of Georgia’s public debt. 

2.  BACKGROUND TO THE INTERVENTION 

Description of the intervention and its objectives 

As described in detail in chapters 2 and 5 of the external evaluation report annexed to 
this  SWD,  Georgia’s  economy  was  heavily  hit  by  an  armed  conflict  with  Russia  in 
August 2008 and the global financial crisis. These severe external shocks, preceded by 
Russian  restrictions  on  some  Georgian  exports  from  2006,  aggravated  the  existing 
internal imbalances and necessitated international financial support to Georgia.  

In  the  external  sector,  Georgia’s  trade  balance  worsened  due  to  weaker  export 
performance,  as  both  the  demand  for  exports  and  commodity  prices  decreased  in  the 
wake of the aforementioned conflict with Russia and the global financial crisis. While 
the  initial  impact  on  the  trade  balance  was  to  some  extent  mitigated  by  an  abrupt 
adjustment  (decrease)  of  import,  Georgia’s  export  was  further  affected  by  a  regional 
economic downturn in  2014-2016 due to  a recession  in  Russia and falling commodity 
prices.  On  the  income  side  of  the  current  account,  Georgia  experienced  a  decrease  in 
primary income (including workers’ remittances) where the balance turned negative in 
2008. 

The  twofold  external  shock  in  2008  also  changed  the  structure  of  Georgia’s  external 
financing.  Debt  financing  increased  significantly  in  2008-2012,  as  opposed  to  foreign 
direct  investment  (FDI)  and  portfolio  equity  inflows.  In  parallel,  the  cost  of  external 
financing  for  Georgia  increased  dramatically,  with  the  implicit  interest  rate  almost 
doubling by 2008 before gradually decreasing in the following years. 

The increase in  debt  financing and the depreciation  of the Georgian lari  in  2014-2016 
resulted in larger external debt. At the same time, Georgia entered the global financial 
crisis  with  a  relatively  low  level  of  foreign  exchange  reserves.  In  2008  and  the 
following few years, international reserves covered neither 3 months of imports nor the 
short-term external debt. 

Regarding the fiscal situation, the Georgian authorities responded to external shocks in 
2008  with  a  countercyclical  policy.  This,  together  with  a  drop  in  revenues,  sharply 
increased the fiscal deficit. A subsequent arrangement with the IMF led to gradual fiscal 
consolidation.  However,  Georgia’s  fiscal  situation  deteriorated  again  in  2014,  mainly 
due to an increase in social spending (for instance, on the Universal Healthcare System 
introduced in 2013). Importantly, Georgia’s fiscal policy relies on periodic realignment 

4 

 
 
of  spending  priorities,  as  revenue  flexibility  is  constrained  by  the  so-called  Economic 
Liberty Act.4 

Georgia’s  public  debt  ratio  to  GDP  increased  on  the  back  of  the  economic  slowdown 
after 2008 and larger fiscal deficits. After a short period of stabilisation, the public debt 
ratio  increased  again  following  the  depreciation  of  the  Georgian  lari  in  2014-2016,  as 
foreign-exchange  denominated  debt  plays  a  dominant  role  in  Georgia’s  public 
financing. 

In this difficult macroeconomic context, Georgia benefitted from several arrangements 
with  the  IMF.  In  September  2008,  the  IMF  approved  an  18-month  Stand-by 
Arrangement  (SBA)  of  SDR  747  million  (USD  1,168  million),  aimed  at  stabilising 
Georgia’s economy and restoring conditions for strong economic growth. Until the last 
drawing in July 2010, SDR 577 million (USD 902 million) were disbursed to Georgia. 
After that, the Georgian authorities treated the arrangement as precautionary and did not 
draw the instalments that became available, despite successful programme reviews. 

After the expiry of this first SBA in 2011, Georgia requested another SBA and a Stand-
by Credit Facility (SCF). In April 2012, the IMF approved a 24-month SBA and SCF 
for  Georgia  with  a  combined  volume  of  SDR  300 million  (USD  464  million).  Again, 
the Georgian authorities treated these arrangements as precautionary and did not borrow 
from the IMF. 

After the expiry of the combined SBA and SCF in April 2014, the Georgian authorities 
requested a further SBA. In July 2014, the IMF approved a 36-month SDR 100 million 
(USD 154 million) SBA with Georgia to support the authorities’ economic programme. 
A  total  of  SDR  100  million  (USD  123  million)  were  disbursed  until  December  2014, 
after the successful completion of the first review. In the course of 2015, the programme 
went  off-track  due  to  disagreements  between  the  between  the  IMF  and  the  Georgian 
authorities  on  banking  supervision,  the  fiscal  strategy  and  the  legal  framework  for 
granting of state guarantees. 

After  these  disagreements  had  been  resolved,  the  IMF  approved  a  36-month  EFF 
programme  of  SDR  210  million  (USD  285  million)  in  April  2017.  This  current 
arrangement with the IMF supports the reform programme of the Georgian authorities, 
with  a  view  to  promoting  higher  and  more  inclusive  growth  while  maintaining 
macroeconomic stability. 

To  complement  Georgia’s  arrangements  with  the  IMF,  the  EU  pledged  two  MFA 
operations of EUR 46 million each at the International Donors’ Conference in Brussels 
in  October  2008.  The  first  of  these  operations  (EUR  46  million,  fully  in  the  form  of 
grants)  was  implemented  in  2009-2010  and  is  outside  the  scope  of  this  evaluation.  In 
view of the continuing residual external  financing need, Georgia requested the second 
operation,  which  is  the  subject  of  this  evaluation,  in  May  2010.  In  January  2011,  the 
Commission proposed an MFA operation of EUR 46 million (half in grants and half in 
loans). The European Parliament and the Council adopted the MFA decision in August 
2013.5  

4   The Act, which came into effect in 2014, caps the  fiscal deficit, public debt and public spending at, 
respectively  3%,  60%  and  30%  of  Gross  Domestic  Product  (GDP)  and  subjects  any  permanent 
increase in taxes (except excise taxes and administrative fees) to a referendum. 

5   A  relatively  long  period  between  the  Commission  proposal  and  the  MFA  decision  of  the  EU  co-
legislators  is  explained  by  procedural  disagreement  between  the  latter  regarding  the  Commission 
proposal for an MFA Framework Regulation. 

5 

 
                                                           
The  MFA  operation  could  not  be  launched  immediately  after  the  adoption  due  to  the 
fact  that  Georgia  did  not  have  a  disbursing,  non-precautionary  arrangement  with  the 
IMF  agreement.  The  approval  of  a  new,  disbursing  IMF  programme  in  July  2014 
allowed the Commission to negotiate with the Georgian authorities a Memorandum of 
Understanding  (MoU)  listing  specific  policy  conditions  for  MFA,  a  Grant  Agreement 
and a Loan Facility Agreement. All three documents were signed in December 2014. 

the  “strengthening 

The general objective of this MFA operation, as stipulated in the MFA decision, was to 
“support  economic  stabilisation  in  conjunction  with  the  current  IMF  programme”. 
transparency  and 
Specific  objectives  were 
accountability, including public finance management systems in Georgia.” In addition, 
in  a  Joint  Declaration  adopted  together  with  the  MFA  decision,  the  EU  co-legislators 
stated that MFA “should aim to restore a sustainable external finance situation” as well 
as “underpin the implementation of a policy programme that contains strong adjustment 
and  structural  reform  measures  designed  to  improve  the  balance  of  payment  position 
[…] and reinforce the implementation of relevant agreements and programmes with the 
Union”. 

the  efficiency, 

Together with the conditionality of the IMF programmes, the MFA operation aimed to 
support further structural reforms in Georgia with a focus on reforms in public finance 
management,  the  social  safety  net,  the  financial  sector  and  the  trade  and  competition 
policies. 

The general intervention logic of MFA operations, which is applicable to the 2015-2017 
Georgia operation, is summarised in Figure 1 below. 

Figure 1. Intervention logic of MFA operations 

6 

 
 
Points of comparison  

The points of comparison, against which the MFA operation of 2015-2017 is assessed, 
refer  to  the  situation  in  Georgia  (1)  prior  to  the  intervention  on  the  one  hand  and  (2) 
during  and  immediately  after  the  implementation  of  the  MFA  operation  on  the  other 
hand.  As  noted  in  the  previous  subsection  which  describes  the  situation  prior  to  the 
MFA operation, particular attention is paid to the external sector, the fiscal situation and 
progress with structural reforms. Georgia’s progress in each of these areas is described 
in Section 3. 

3. 

IMPLEMENTATION / STATE OF PLAY 

Economic situation  

This  subsection  describes  the  economic  situation  in  Georgia  during  and  immediately 
after  the  implementation  of  the  MFA  operation  of  2015-2017,  with  reference  to  the 
points of comparison in the external sector and the fiscal situation. The implementation 
of this operation itself, including Georgia’s progress on structural reforms supported by 
MFA policy conditionality, is described in the subsequent subsection. 

As described in detail in chapter 5 of the external evaluation report, Georgia’s balance-
of-payments situation gradually improved after the regional slowdown in 2014-2016. 
The  trade  deficit  decreased  starting  from  2016,  supported  by  strong  export  growth. 
Together with gradual recovery in remittances, the narrowing of the trade deficit helped 
to  decrease  the  current  account  deficit.  These  developments  relating  to  Georgia’s 
current account and remittances are summarised in, respectively, Figures 2 and 3 below. 

Figure 2. Structure of the current account (% of GDP) 

7 

-35-30-25-20-15-10-5051015-35-30-25-20-15-10-505101520042005200620072008200920102011201220132014201520162017Percent of GDPPercent of GDPNet exportSecondary incomePrimary incomeCapital accountCurrent accountNet external position 
 
 
 
 
 
 
 
 
 
 
Figure 3. Remittances (% of GDP) 

In  the  financial  account,  the  non-debt  financing  (FDI  and  portfolio  investment) 
recovered  starting  from  2014.  At  the  same,  the  cost  of  external  financing  decreased 
from 2016. These developments are summarised below in Figures 4 and 5, respectively. 

Figure 4. External financing (% of GDP) 

8 

02468101220042005200620072008200920102011201220132014201520162017Remittances (in percent of  GDP)Remittances: CreditNet remittances-10-50510152025-10-505101520252 0002 0012 0022 0032 0042 0052 0062 0072 0082 0092 0102 0112 0122 0132 0142 0152 0162 017Percent of GDPPercent of GDPNon-debt financing**Debt financingFX Reserves***Net external borrowing* 
 
  
 
 
 
 
 
 
 
 
 
Figure 5. Implicit financing cost6 of external liabilities (%) 

Georgia’s  external  debt  continued  to  increase,  as  shown  in  Figure  6.  The  main  factor 
behind  the  increase  is  the  depreciation  of  the  national  currency  in  2015-2016. 
Vulnerabilities relating to high level of external debt are to some extent mitigated by the 
facts  that  around  80%  of  the  stock  is  long-term  debt  and  that  the  cost  of  financing 
remains relatively low, reflecting a significant share of concessional financing. On the 
other hand, around 90% of Georgia’s external debt is denominated in foreign currency, 
which points to a significant foreign exchange risk. 

Figure 6. External debt (% of GDP) 

Georgia’s  foreign  exchange  reserves  increased  over  the  reference  period  (2015-2017) 
but remained below the level considered to be adequate, as shown in Figure 7. 

6   Implicit financing cost is calculated as the ratio of interest payments on debt liabilities to the stock of 
debt (excluding intercompany loans). It shows the average financing cost of the outstanding debt. 

9 

2.02.53.03.54.04.55.05.56.020012002200320042005200620072008200920102011201220132014201520162017PercentLoansExternal debtExternal debt incl. intercompany loans0204060801001201401600204060801001201401601999200020012002200320042005200620072008200920102011201220132014201520162017Percent of GDPPercent of GDPExternal debt and liability indicatorsNet external debtNon-debt type liabilities*Net foreign liabilitiesGross external debt 
 
 
 
 
 
 
                                                           
Figure 7. International reserves (USD billion and the IMF ARA7 metric) 

In  terms  of  the  fiscal  situation,  the  deficit  increased  slightly  during  the  reference 
period, as shown in Figure 8. This increase mainly reflects higher government spending 
(notably,  social  spending,  including  healthcare)  and  was  one  of  the  reasons  why  the 
IMF SBA of 2014 went off-track. As part of the agreement on a new IMF programme in 
2017,  the  Georgian  authorities  committed  to  fiscal  consolidation  that  resulted  in 
subsequent gradual reduction of the deficit. 

Figure 8. Fiscal deficit (% of GDP) 

Over  the  reference  period,  the  Georgian  authorities  also  stabilised  and  then  started  to 
reduce the level of public debt (Figure 9). These public debt dynamics are mainly due to 
limited public financing needs (which reflect a prudent fiscal policy) and GDP growth 
(averaging in 3.5% in 2015-2017). 

7   Assessment 

of 

Reserve 

Adequacy.  More 

information 

is 

available 

at: 

https://www.imf.org/external/np/spr/ara/. 

10 

0.00.20.40.60.81.01.21.41.60.00.51.01.52.02.53.03.54.02006200720082009201020112012201320142015201620172018Compared to ARA MetricIn Billion USDFX reserves adequacy based on IMF ARA metricsExport revenues (5 % weight)Broad money (5% weight)Short-term Debt (30% weight)Other Liabilities (15% weight)Reserves, USD billionReserves ARA Metric (RHS)-10-8-6-4-2024-10-8-6-4-202420042005200620072008200920102011201220132014201520162017Percent of GDPPercent of GDPCyclically adjusted primary balance*Cyclical component*InterestPrimary balanceGovernment balance 
 
 
 
 
 
                                                           
Figure 9. Public debt (% of GDP) 

Implementation of the MFA operation  

After  the  entry  into  force  of  the  MoU,  the  Grant  Agreement  and  the  Loan  Facility 
Agreement, the EU disbursed to Georgia the first instalment (EUR 13 million in grants 
and  EUR  10 million  in  loans)  of  the  MFA  operation  under  evaluation.  The  grant 
component  was  disbursed  in  January  2015  and  the  loan  component  in  April  2015 
(allowing the EU to obtain more favourable financing conditions from the market and 
pass them on to Georgia). As also required for the first disbursement, Georgia continued 
to  meet  the  political  precondition  for  MFA  (respect  for  effective  democratic 
mechanisms,  rule  of  law  and  human  rights)  and  make  good  progress  with  the 
implementation of the IMF programme. 

For  the  second  instalment  (EUR  10 million  in  grants  and  EUR  13 million  in  loans), 
Georgia  was  additionally  required  to  implement  specific  policy  conditions  set  in  the 
MoU that supported structural reforms. The second instalment could not be disbursed 
as planned in  2015, despite the fact  that Georgia implemented these policy  conditions 
(as assessed during a review mission of the Commission staff in November 2015) and 
continued to meet the political pre-condition for MFA. As explained in Section 2, this 
was  because  the  IMF  SBA  of  2014  running  off-track.  The  second  instalment  was 
disbursed in May 2017, after the Georgian authorities resolved disagreements with the 
IMF and agreed on a new programme (EFF) in April 2017. 

In  the  area  of  public  finance  management,  MFA  policy  conditionality  has  helped  to 
establish  a  dedicated  training  centre,  in  order  to  improve  the  functioning  of  the 
electronic  public  procurement  system,  as  well  as  to  strengthen  the  operational 
independence  of  the  State  Audit  Office.  Regarding  the  social  safety  net,  MFA 
conditions have helped to improve the management of the Universal Healthcare System 
and  increase  the  quality  of  healthcare  services  by  ensuring,  respectively,  that  the 
Georgian  authorities  complete  a  spending  survey  and  establish  a  quality  management 
unit  at  the  relevant  ministry.  In  the  financial  sector,8  MFA  conditions  have  supported 
the strengthening of capital adequacy in Georgian banks and risk management capacity 

8   For further detail on structural reforms in the financial sector, please refer to the relevant case study 

in Annex VII of the external evaluation report. 

11 

0102030405060702002200320042005200620072008200920102011201220132014201520162017Percentof GDPForeign DebtDomestic DebtGovernment  Debt 
 
 
                                                           
at  the  National  Bank  of  Georgia.  In  terms  of  trade9  and  competition  policies,  MFA 
conditions  have  supported  export  performance  and  strengthened  the  competition 
regulation  by  ensuring,  respectively,  that  the  authorities  centralise  the  issuance  of 
certificates of origin and adopt the necessary secondary legislation.  

The  Georgian  authorities  were  effective  in  the  implementation  of  the  conditionality, 
meeting  all  of  them  without  the  need  for  waivers.  On  the  EU’s  side,  the  monitoring 
process  was  adequate  and  the  Commission  complied  with  all  checks  ensuring  that 
Georgia  had  satisfactory  fulfilled  the  reform  measures  supported  by  the  MFA.  
Implementation of MFA policy conditions is assessed in detail in Section 5, as part of 
the relevance and effectiveness of the MFA operation of 2015-2017. 

Continuing  to  face  external  weaknesses,  the  Georgian  authorities  requested  further 
MFA from the Commission in June 2017. The proposal of a third MFA in the amount of 
EUR 45 million (EUR 35 million in loans and EUR 10 million in grants) was adopted 
by the Commission on 29 September 2017, and approved by co-legislators on 18 April 
201810.  The  Commission  has  disbursed  the  first  instalment  (EUR  15  million  loan  and 
EUR  5  million  grant)  to  Georgia  in  December  2018.  The  second  and  final  instalment 
can  be  disbursed  in  2019,  provided  that  the  IMF  programme  remains  on  track  and 
Georgia meets the agreed policy conditions. 

4.  METHODOLOGY 

Evaluation techniques used 

The  methodology  for  evaluating  the  MFA  operation  in  Georgia  of  2015-2017  was 
guided by the Commission’s Better Regulation Guidelines11 and the Guidelines for the 
Ex-Post evaluation of Macro-Financial Assistance Operations.12 

This  evaluation  was  supported  by  an  assessment  carried  out  by  an  external  contractor 
from  September 2018 until  April 2019 and was  overseen by an  Inter-Service Steering 
Group  (ISG).  The  external  evaluation    comprised  of  the  following  four  phases:  (1) 
inception, (2) data collection, (3) analysis, and (4) finalisation. 

Techniques used for the evaluation are summarised in Figure 10 and described below. 
These techniques  are  closely linked to  the consultation  strategy, which is  described in 
Annex  2  and  includes  the  synopsis  of  results.  Further  information  on  the  evaluation 
techniques  is  available  in  Chapter  4  of  the  external  evaluation  report.  Overall, 
triangulation  of  findings  obtained  using  different  techniques  has  helped  to  increase 
validity of the evaluation results. 

9   For further detail on structural reforms in the area of trade policy, please  refer to the relevant case 

study in Annex VII of the external evaluation report. 
10    Decision (EU) 2018/598 (OJ L 103, 23.4.2018, p. 8–13) 

11   Available at: http://ec.europa.eu/smart-regulation/guidelines/docs/swd_br_guidelines_en.pdf. 
12   Available at: 

http://ec.europa.eu/dgs/economy_finance/procurement_grants/calls_for_tender/2015/015d/annex4-
methodological_orientations_en.pdf. 

12 

 
 
                                                           
Figure 10. Evaluation techniques 

Evaluation approach – Triangulation 

Triangulation of findings from 
different evaluation methods: 

a.  Desk research: 

-  Document analysis; 

-  Quantitative analysis. 

b.  Semi-structured interviews; 

c.  Two focus groups: 

- 

Structural and Social 

Reforms; 

-  Macroeconomic, fiscal 

and financial sector. 

d.  Delphi survey to key 

experts. 

The desk research involved document analysis and quantitative analysis. 

The  main  documents  used  for  the  evaluation  were  the  MFA  Decision,  the  SWD 
accompanying  the  Commission  proposal,13  and  implementation  reports  drafted  by 
Commission’s  Directorate-General  for  Economic  and  Financial  Affairs  (DG  ECFIN). 
Additional sources included other EU documents (such as the EU-Georgia Association 
Agreement),  reports  of  the  IMF,  World  Bank  and  other  international  organisations,  as 
well as analysis carried out by research institutes. 

For  quantitative  analysis,  the  evaluation  used  data  from  the  national  sources  (mainly, 
Geostat,  the  National  Bank  of  Georgia  and  the  Ministry  of  Finance)  as  well  as  from 
international  organisations  such  as  the  IMF  and  the  World  Bank.  The  quantitative 
analysis  notably  covered  macroeconomic  fundamentals,  fiscal  indicators,  external 
sustainability variables, financial sector variables, and structural reforms (e.g. variables 
measuring  institutional  development).  Additionally,  a  macroeconomic  tool  developed 
by  the  IMF  was  used  for  debt  sustainability  analysis  (DSA),14  while  fiscal  savings 
(resulting  from  concessional  terms  of  the  MFA  operation  in  question,  as  compared  to 
market-based  alternatives)  were  calculated  by  comparing  the  face  value  and  the  net 
present value of the operation. 

The  rest  of  the  techniques  –  semi-structured  interviews,  focus  groups  and  a  Delphi 
survey – fall into a category of qualitative analysis. 

13   Ex-ante  evaluation  statement  on  further  macro-financial  assistance  to  Georgia  (COM(2010)  804 

final). 

14   IMF Staff Guidance Note for Debt Sustainability Analysis in Market-Access Countries, available at: 
https://www.imf.org/external/np/pp/eng/2013/050913.pdf. For the methodology of DSA calculations 
in the evaluation of the MFA operation 2015-2017 in Georgia, please also refer to Annex VI of the 
external evaluation report. 

13 

 
 
 
                                                           
The  semi-structured  interviews15  aimed  at  gathering  information  on  the  design, 
implementation  and  results  of  the  MFA  operation  of  2015-2017,  with  a  focus  on  the 
macroeconomic and fiscal situation in Georgia, structural reforms, the social impact and 
debt  sustainability. The interviews were confidential,  and pre-interview questionnaires 
were  used  to  improve  the  quality.  More  than  60  stakeholders  were  interviewed.  They 
included representatives of the Georgian authorities, international financial institutions, 
as well as donors and external actors (such as experts of economic research institutes). 
In  addition,  interviews  were  carried  out  with  the  European  Commission  (both  in  the 
headquarters and in the EU Delegation to Georgia). 

Meetings  of  two  focus  groups  were  organised  in  Tbilisi,  covering  (1)  structural  and 
social  reforms  in  Georgia  as  well  as  the  relevance  of  MFA  policy  conditions,  and  (2) 
macroeconomic  and  fiscal  developments.  Each  focus  group  was  formed  of  eleven 
participants, but while the first group mainly involved researchers and representatives of 
non-governmental  organisations,  the  second  group  targeted  representatives  of  the 
financial sector. 

A Delphi survey was undertaken with a panel of 16 experts. Experts were interviewed 
using a structured questionnaire and were then invited to participate in a second round 
to  reconsider their positions in  light  of other respondents’ opinions. The results of the 
Delphi survey, which focussed on the value added of the MFA operation of 2015-2017, 
are summarised in Annex 2.16 

Risks and limitations 

While  the  overall  reliability  and  validity  of  the  evaluation  is  strong,  a  number  of 
methodological limitations and challenges effected the evaluation:  

  While generally the data coverage is good, it is limited in some areas of interest 
(notably,  as  regards  social  indicators)  and  longer  time  series  are  not  always 
available.  An  additional  limitation  relates  to  fiscal  statistics.  The  Georgian 
authorities  publish  several  measures  of  fiscal  deficit,  without  sufficiently 
explaining the methodological differences. 

  Another limitation relates to the timing of the MFA operation in question. The 
operation was implemented relatively long after its inception, meaning that the 
rather  different 
inception  and 
macroeconomic circumstances, which complicates the assessment.  

took  place  under 

implementation 

the 

  This changing economic environment over an extended period and the fact that 
the  MFA  operation  was  implemented  in  parallel  with  IMF  and  other 
international  support  programmes  also  makes  it  difficult  to  disentangle  the 
impacts of the MFA operation from the impacts resulting from other factors. 
  Regarding  evaluation  techniques,  in  response  to  an  initial  low  up-take,  the 
Delphi panel was broadened. This might have resulted in the inclusion of several 
experts  with  less  detailed  knowledge  of  the  MFA  operation,  as  evidenced  in  a 
relatively high proportion of “no opinion” answers. 

Nonetheless,  the  identified  risks  and  limitations  do  not  put  into  question  the  overall 
reliability  of  the  evaluation,  and  are  mitigated  by  the  fact  that  results  were  obtained 

15   See the list of completed interviews in Annex II of the external evaluation report. 
16   Detailed results are available in Annex IV of the external evaluation report. 

14 

 
 
                                                           
using  different  evaluation  techniques.  Therefore,  the  conclusions  reached  in  the 
evaluation can be considered as valid. 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS 

Evaluation Framework 

The  evaluation  covered  five  evaluation  criteria  used  in  the  assessment  of  EU 
programmes,  namely:  (1)  relevance,  (2)  effectiveness,  (3)  efficiency,  (4)  EU  added 
value,  and  (5)  coherence.  The  evaluation  considered  two  additional  criteria:  (6)  social 
impact and (7) debt sustainability. 

Answers to questions17 

Evaluation Question 1: Relevance of the operation  

To what extent were the MFA operation design and outcomes appropriate in 
relation to the outputs to be produced and the objectives to be achieved? 

1.1. To what extent can the MFA design and outcomes be considered to have been 
appropriate? 

1.2. Were the amounts and terms of the financial assistance provided to Georgia 
adequate?  

1.3. Was the conditionality of the MFA operation appropriate in relation to the 
objectives to be achieved? 

1.4. How did the long timeline of the MFA operation impact its relevance? 

Question  1.1  refers  to  the  relevance  of  objectives  and  the  overall  design  of  the  MFA 
operation under evaluation.  

Based on the MFA Decision and the accompanying Joint Declaration and as described 
in Section 2, the objectives of the MFA operation were, essentially, to alleviate short-
term  external  financing  pressure  and  help  Georgia  return  to  a  sustainable  path.  These 
objectives were relevant in 2011 when the Commission submitted its proposal to the EU 
co-legislators.  As  described  in  Section  3,  Georgia  was  facing  external  vulnerabilities 
such as an increasing current account deficit, subdued non-debt financing and high level 
of  foreign  currency  denominated  debt.  Stakeholders  consulted  during  this  evaluation 
have confirmed these vulnerabilities.  

The  overall  design  of  the  MFA  operation  was  also  relevant  to  its  objectives.  More 
specifically,  the  criteria  for  the  first  instalment  (ratification  of  the  MoU  and  the  IMF 
programme being on track) allowed for a swift disbursement. This was in line with the 
objective to provide short-term relief. The second instalment was additionally subject to 
the  implementation  of  several  policy  conditions,  which  provided  a  suitable  means  for 
encouraging  structural  reforms.  A  majority  of  respondents  in  the  Delphi  survey  have 
confirmed  that  this  design  (subjecting  only  the  second  disbursement  to  policy 
conditionality) was the optimal choice.  

Question 1.2 refers to the amount and terms of the MFA operation of 2015-2017. 

Georgia’s  external  financing  gap  for  the  period  of  2009-2011  was  assessed  in  the 
Commission SWD accompanying the proposal for the MFA under evaluation to the EU 

17   For more detail, please refer to Chapter 6 of the external evaluation report. 

15 

 
 
 
                                                           
co-legislators. The amount of the EU contribution in the form of two MFA operations, 
pledged  at  the  International  Donors’  Conference  in  Brussels  in  2008,  covered  14%  of 
the  residual  financing  gap.18  The  EU  contribution,  which  would  have  closed  the 
estimated residual financing gap, was considered to be appropriate (in terms of burden 
sharing  with  other  donors)  and  proportional  (limited  to  the  minimum  necessary  to 
achieve short-term macroeconomic stability in Georgia).  

Despite a long  gestation of the MFA operation  under evaluation, its amount  (EUR 46 
million)19 was still relevant at the time of the actual disbursements. The first instalment 
was disbursed in January and April 2015 when the external financing needs of Georgia 
(as  reassessed  by  the  Commission  and  the  IMF  in  2014)  were  similar  to  the  level 
estimated  in  2011.  As  for  the  second  instalment  that  was  disbursed  in  May  2017,  it 
helped to close the financing gap identified as part of the EFF programme agreed with 
the IMF staff in April 2017, mostly in form of replenishing Georgia’s foreign exchange 
reserves. 

Regarding the terms of the MFA operation of 2015-2017, half of the amount (EUR 23 
million)  was  provided  in  grants  and  another  half  in  loans.  This  particular  proportion 
reflected concerns about the level of Georgia’s public debt (arguing in favour of a grant) 
and  the  country’s  eligibility  for  concessional  financing  from  international  financial 
institutions  such  as  the  World  Bank.  At  the  same  time,  the  EU  took  into  account  an 
improvement in Georgia’s economic situation and the level of development since 2008. 
Hence,  a  grant  element  was  maintained  but  also  reduced,  compared  to  the  previous 
MFA operation that was fully in the form of grants.20 Stakeholders have acknowledged 
this coherence during the interviews. The results of the Delphi survey have been more 
diverse:  half  of  the  respondents  thought  that  an  MFA  operation  fully  in  the  form  of 
grants would have contributed more to addressing Georgia’s financing needs, while also 
acknowledging the disciplining effect of loans.  

Question  1.3  refers  to  the  relevance  of  MFA  policy  conditions  in  relation  to  the 
objectives of the operation, as well as to the design of these conditions. 

As  confirmed  by  the  stakeholder  interviews  and  the  first  focus  group  on  structural 
reforms, the four areas of MFA policy conditionality – public finance management, the 
financial sector, the social safety net, and the trade and competition policies – covered 
the  most  relevant  reform  challenges  in  Georgia.  A  majority  of  the  respondents  in  the 
Delphi  survey  have  also  confirmed  that  all  the  eight  MFA  policy  conditions  were 
important for structural reforms in the country. Additionally, several conditions (such as 
strengthening  risk  management  in  the  banking  sector)  were  highly  relevant  for  the 
macroeconomic stabilisation objective of the MFA operation under evaluation.  

According to several stakeholders, some additional measures could have been relevant 
as MFA policy conditions. Specific examples refer to strengthening the capacity of the 
Social Services Agency that manages the Universal Healthcare System and supporting 
responsible lending practices in the banking sector. The Commission acknowledges the 
relevance  of  these  issues,  while  also  being  aware  that  the  list  of  conditions  has  to  be 
focused  and  limited  (as  also  recognised  by  the  stakeholders;  see  below).  In  fact,  the 

18   Gap remaining after the contributions from the IMF and the World Bank. 
19   0.4% of Georgia’s GDP in 2015 (the first year of implementation of the MFA operation). 
20   The  trend  of  gradually  reducing  the  grant  element  continues,  in  line  with  the  improvement  in 
Georgia’s  situation.  In  the  MFA  operation  approved  in  2018,  EUR  10  million  (or  22%)  out  of  the 
total amount of EUR 45 million are in the form of grants. 

16 

 
                                                           
Commission  has  taken  up  some  of  them21  as  policy  conditions  for  the  current  MFA 
operation approved in 2018.  

Based  on  the  responses  in  the  Delphi  survey,  the  policy  conditions  of  the  MFA 
operation under evaluation were well designed, both in terms of their mix and in terms 
of their number. The in-built flexibility of some conditions as well as the fact that they 
were designed to be implemented swiftly after the MoU had been negotiated meant that 
there was no need for further flexibility during the implementation phase. At the same 
time, the in-built flexibility of some relevant measures to be implemented swiftly also 
meant  that  they  had  to  be  defined  in  less  precise  terms  (e.g.  the  condition  on  the 
establishment of a healthcare quality management unit). 

The  impact  of  the  timeline  (from  proposal  to  disbursement)  of  the  operation  on  its 
relevance  (Question  1.4)  was  limited.  The  objectives  of  macroeconomic  stabilisation 
and  supporting  structural  reforms  were  just  as  relevant  in  2014  (when  the  MoU  was 
negotiated and the implementation started) as in 2011 (when the Commission submitted 
its proposal to the EU co-legislators). In terms of macroeconomic challenges, Georgia 
was  facing  a  regional  crisis  that  affected  both  its  balance-of-payments  and  fiscal 
situation. In terms of structural reforms, the MFA operation under evaluation not only 
supported them directly during a difficult period but also (indirectly) created space for 
reforms by easing the macroeconomic adjustment. These views have been confirmed by 
the Delphi survey and the stakeholder interviews. 

Evaluation Question 2: Effectiveness  

To what extent have the objectives of the MFA operation been achieved? 

2.1. To what extent has the MFA operation been effective in improving macroeconomic 
conditions (with focus on the Balance of Payments and exchange rate)? 

2.2. To what extent has the MFA operation been effective in terms of fiscal 
consolidation? 

2.3. To what extent have the short and medium-term expected structural effects of the 
assistance occurred as envisaged? 

In order to  assess how the MFA operation of 2015-2017 helped to  improve Georgia’s 
balance-of-payments  situation  (Question  2.1),  the  evaluation  considers  the  external 
financing situation and confidence in the economy (as manifested in the market-based 
financing cost, exchange rates and dollarisation). 

To  quantify  the  effect  of  the  MFA  operation  under  evaluation  on  Georgia’s  external 
debt  sustainability  in  the  medium  (2014-2019)  and  long  (beyond  2019)  term,  three 
scenarios have been computed, using the aforementioned IMF’s DSA framework. The 
first scenario (A) is the baseline and incorporates the (actual) effects of both the MFA 
operation of 2015-2017 and the IMF SBA approved in 2014. The second (alternative) 
scenario (B) assumes that neither the MFA nor the IMF SBA were available to Georgia. 
Finally,  the  third  (alternative)  scenario  (C)  assumes  that  only  the  IMF  SBA  was 
available. 

21   The  capacity  of  the  Social  Services  Agency  is  being  strengthened  as  part  of  a  policy  condition  on 
selective  contracting,  attached  to  the  disbursement  of  the  second  instalment  of  the  MFA  operation 
approved in 2018. 

17 

 
 
                                                           
As shown in Figure 11, the combined impact of the MFA and the IMF SBA (baseline, 
scenario A) helped to  limit  the increase in  Georgia’s  external  debt.  A similar, if more 
limited, impact is noticeable in the alternative scenario C, assuming that only the IMF 
operation was  available  to  Georgia. This  latter,  more limited impact  is  explained by  a 
smaller  size  of  the  MFA  operation,  compared  to  the  IMF  SBA.  The  aforementioned 
effects are likewise observable in the long-term, meaning that the MFA operation under 
evaluation helped to ease external financing pressures in Georgia, as also confirmed by 
the  Delphi  survey.  At  the  same  time,  this  long-term  trend  will  depend  on  Georgia’s 
ability to address structural issues, e.g. to increase productivity and trade integration.  

Figure 11. Projections of Georgia’s external debt-to-GDP ratio (%) 

In  terms  of  confidence  in  Georgia’s  economy,  the  combined  impact  of  the  MFA 
operation of 2015-2017 and the IMF SBA was also positive. As shown in Figure 12, the 
external financing cost – measured by the spreads of the Georgian Eurobond over a the 
US  benchmark  yield  and  the  JP  Morgan  Emerging  Market  Bond  Index  (EMBI)  – 
decreased in mid-2014 when the IMF SBA was approved and negotiations on the MoU 
of the MFA operation in question restarted.  

Figure  12.  Spread  of  a  Georgian  Eurobond  yield  over  the  US  benchmark  and  the  EMBI 
(percentage points) 

18 

60657075808590951001052013201520172019202120232025202720292031203320352037203920412043204520472049Percent of GDPScenario AScenario BScenario C-2-1012345Jan 2012May 2012Sep 2012Jan 2013May 2013Sep 2013Jan 2014May 2014Sep 2014Jan 2015May 2015Sep 2015Jan 2016May 2016Sep 2016Jan 2017May 2017Sep 2017Jan 2018Spread (over EMBI)Spread (over US bond) 
 
 
 
Similar impacts are also observable in relation to the exchange rates (with the Georgian 
lari appreciating against the euro and the US dollar in mid-2014) and the dollarisation 
(the  increased  confidence  being  reflected  in  lower  deposit  dollarisation).22  These 
developments are summarised below in Figure 13 and Figure 14 respectively.  

Figure 13. Euro and the US dollar exchange rates against the Georgian lari 

Figure 14. Deposit and loan dollarisation in Georgia (%) 

It is true, however, that the aforementioned trends relating to the external financing cost, 
exchange rates and dollarisation have been to some extent reversed in 2015-2016, due to 
a  regional  economic  slowdown  and  the  accommodative  monetary  policy  that  the 
National Bank of Georgia pursued in response.  

Question  2.2  refers  to  the  effectiveness  of  the  MFA  operation  of  2015-2017  in 
supporting fiscal consolidation in Georgia. 

The  undesignated  nature  of  MFA  and  the  fact  that  the  operation  in  question  was 
disbursed  on  highly  concessional  terms  (partly  in  grants  and  partly  in  concessional 

22   Deposit  dollarisation  is  more  responsive  to  changes  in  confidence  due  to  the  fact  that  maturity  of 

deposits is typically shorter than that of loans. 

19 

1.51.71.92.12.32.52.72.91.92.12.32.52.72.93.13.3Jan-2012Jun-2012Nov-2012Apr-2013Sep-2013Feb-2014Jul-2014Dec-2014May-2015Oct-2015Mar-2016Aug-2016Jan-2017Jun-2017Nov-2017Apr-2018Sep-2018USD/GELEUR/GELMonthly average Euro and USD exchange ratesEURO (LHS)US Dollar (RHS)5055606570752012Q12012Q22012Q32012Q42013Q12013Q22013Q32013Q42014Q12014Q22014Q32014Q42015Q12015Q22015Q32015Q42016Q12016Q22016Q32016Q42017Q12017Q22017Q32017Q42018Q12018Q22018Q32018Q4Dollarisation (percent)DepositsLoans 
 
 
 
                                                           
loans)  had  a  direct,  if  limited,  positive  impact  on  the  fiscal  position  of  Georgia.23 
Through this impact, the MFA operation of 2015-2017 helped to smooth the adjustment 
path and create fiscal space for reforms and sustained social spending.  

Aside  from  this  short-term  impact,  some  policy  conditions  attached  to  the  MFA 
operation  had  a  longer-term  positive  impact  on  Georgia’s  public  finances.  This  was 
particularly  true  of  conditions  that  helped  to  improve  public  procurement  and  the 
efficiency  of  healthcare  spending.  The  stakeholder  interviews  and  the  Delphi  survey 
have confirmed the view that, while the MFA operation had a limited short-term fiscal 
impact,  its  biggest  contribution  in  terms  of  public  finances  was  through  structural 
reforms.  

The fiscal consolidation itself took place in the context of programmes agreed with the 
IMF  and  complemented  by  MFA.  While  the  original  arrangement  (SBA  approved  in 
2014) went off-track in 2015, partly because of the above-target fiscal deficit, Georgia 
returned to a consolidation path in late 2016. This allowed reaching an agreement on a 
new  IMF  programme  (EFF  approved  in  2017),  which  provided  for  further  gradual 
consolidation. 

The  short  and  medium-term  structural  effects  of  the  MFA  operation  of  2015-2017 
(Question  2.3)  largely  materialised  as  planned.  The  main  reason  for  this  was  the  fact 
that  Georgia  satisfactorily  implemented  all  policy  conditions  attached  to  the  MFA 
operation, as also confirmed by the stakeholders interviews and the Delphi survey. The 
latter  has  also  confirmed  that  MFA  conditionality  helped  to  implement  the  relevant 
reforms faster than would have been the case without the MFA operation.  

More specifically, the public procurement training centre and the relevant  certification 
system  were  established  in  2014.  This  has  helped  to  improve  the  functioning  of  the 
electronic public procurement system, as also recognised in the Public Expenditure and 
Financial Accountability Report of 2018.24 Likewise, the Parliament’s attitude towards 
the  State  Audit  Office  (SAO)  has  improved,  despite  the  fact  that  the  SAO  was 
occasionally  required  to  provide  data  that  goes  beyond  its  annual  financial  audit.  The 
Healthcare Management and Utilisation Survey was presented in 2015 and the tasks of a 
healthcare quality management unit were assigned in 2014. These reforms have helped 
to  improve  the  efficiency  of  the  Universal  Healthcare  System,  although  its  financial 
sustainability  remains  a  concern.25  In  the  financial  sector,  Georgian  banks  submitted 
their Internal Capital Adequacy Assessment Process (ICAAP) reports in 2014, while the 
National  Bank  of  Georgia  completed  the  Supervisory  Review  and  Evaluation  Process 
(SREP) of two largest banks and established a centralised risk management department 
in, respectively 2015 and 2014. These structural changes have ensured resilience of the 
Georgian  banking  sector  despite  a  regional  economic  downturn  of  2014-2016.  The 
issuance of certificates of origin  (so-called EUR  1 certificates) has been  centralised  at 
the  Ministry  of  Finance,  reducing  the  regulatory  burden  for  exporters.  Finally,  the 
secondary legislation relating to the Law on Competition was adopted in 2014, allowing 
Georgia to establish the Competition Agency that same year. 

23   Fiscal  savings  attributable  to  the  MFA  operation  of  2015-2017  are  discussed  in  response  to 

evaluation question 7. 

24   Available 

https://pefa.org/sites/default/files/GE-Jun18-PFMPR-
Public%20with%20PEFA%20Check.pdf.  Some  remaining  weaknesses  (e.g.  relating  to  the  review 
process and a high proportion of direct procurement) are being addressed as part policy conditionality 
of the current MFA operation approved in 2018. 

at: 

25   As  noted,  this  concern  is  being  addressed  as  part  of  a  policy  condition  on  selective  contracting, 

attached to the current MFA operation approved in 2018. 

20 

 
                                                           
The MFA operation in question also had less direct structural effects, by complementing 
other  international  support  programmes  (as  discussed  in  response  to  the  evaluation 
question 4 below). 

Evaluation Question 3: Efficiency of the operation 

To what extent did the MFA operation design and implementation allow to carry 
out the intervention efficiently? 

3.1. In what way has the design of the MFA assistance conditioned the performance of 
the operation in respect to its costs and its objectives? 

3.2. How did the long timeline of the MFA operation impact its efficiency? 

3.3. Was  the disbursement of the financial assistance appropriate in  the context  of the 
prevailing economic and financial conditions in Georgia? 

The design of the MFA operation in question ensured efficient performance in relation 
to costs and objectives (Question 3.1).  

From the perspective of the EU, efficiency is evident from a relatively low cost of the 
MFA  operation,  compared  to  its  significant  impact.  Whilst  the  cost  to  the  EU  budget 
was limited to the amount of MFA grants provided, the significant impact came about 
from  the  mutual  leverage  of  the  MFA  operation  and  other  international  support 
programmes such as those of the  IMF, which MFA complements26. The design of the 
MFA operation also ensured synergies with other EU policies and instruments, notably 
with  cooperation assistance operations in  the areas of public finance management  and 
Georgia’s  economic  integration  with  the  EU.27  In  this  sense,  the  MFA  operation  of 
2015-2017  was  more  cost-efficient  than  would  have  been  the  case  if  various  EU 
Member States had provided the same amount of assistance to Georgia individually. 

From the perspective of the Georgian authorities, policy measures required as part of the 
MFA operation under evaluation entailed a direct cost. However, the additional element 
of this cost was limited: many of these policy measures were part of the national reform 
agenda  anyway  and,  in  some  cases,  were  also  required  by  other  international  support 
programmes.  In  terms  of  the  financial  impact,  the  Georgian  authorities  would  have 
benefitted more if the whole MFA amount had been provided in the form of grants. At 
the same time, the highly concessional terms of the loan element (a 15-year maturity, a 
10-year grace period and an effective weighted average interest rate of 0.88 per cent)28 
were more attractive than alternative financing options. 

As  already  discussed  in  response  to  evaluation  question  1.4  above,  disbursements  of 
MFA were appropriate in the given circumstances, despite the relatively long  timeline 
of the operation (Questions 3.1 and 3.2). At the time of the first disbursement (of a grant 
and  a  loan  in,  respectively  January  and  April  2015),  Georgia  was  facing  a  difficult 

26   For  the  value  added  in  relation  to  assistance  by  other  donors,  please  refer  to  the  response  to  the 

evaluation question 4. 

27   For the coherence with other EU external actions towards Georgia, please refer to the response to the 

evaluation question 5. 

28   For  the  impact  on  the  sustainability  on  Georgia’s  public  debt,  please  refer  to  the  response  to  the 

evaluation question 7. 

21 

 
 
                                                           
balance-of-payments  and  fiscal  situation  due  to  a  regional  crisis.  At  the  time  of  the 
second disbursement, Georgia needed to increase inadequate foreign exchange reserves. 

The  approval  of  the  MFA  operation  in  question  took  relatively  long  (more  than  three 
years  from  the  official  request  by  the  Georgian  authorities).  This  was  partly  due  to 
procedural disagreements between the European Parliament and the Council regarding a 
framework  regulation  for  MFA,  which  put  on  hold  the  approval  of  all  new  MFA 
operations. At the same time, the resulting efficiency loss was minimal because of the 
absence  of  a  disbursing  IMF  programme  (one  of  the  conditions  for  MFA)  from  late 
2010 until April 2014 and from mid-2015 until April 2017. This would have prevented 
disbursements  of  MFA  even  if  the  operation  under  evaluation  had  been  approved 
earlier. 

The  external  evaluators  have  pointed  out  that  the  period  from  the  time  when  the 
Georgian authorities submitted the official request for MFA (May 2010) until the time 
when  the  Commission  submitted  its  proposal  to  the  EU  co-legislators  (January  2011) 
could have been shorter, given that MFA is a rapid response instrument. However, the 
previous  MFA-I  operation  was  only  fully  disbursed  and  completed  in  August  2010, 
before  which  the  Commission  could  not  start  preparing  a  new  proposal.  Upon  the 
completion of the MFA-I to Georgia, the Commission  prepared the new  proposal  and 
adopted it in January 2011. In effect therefore, it took the Commission four months to 
prepare and adopt the new MFA proposal. The Commission considers this to be within 
a normal timeframe, yet will endeavour to shorten this period whenever possible, as has 
already been the case in several subsequent proposals. 

Evaluation Question 4: EU added-value of the operation 

What  was  the  rationale  for  an  intervention  at  EU  level?  To  what  extent  did  the 
MFA operation add value compared to other interventions by other international 
donors? 

4.1. To what extent have the expected benefits of the EU intervention been attained? 

4.2.  What  is  the  value  resulting  from  the  EU  assistance  which  is  additional  to  the 
assistance obtained at other levels (IMF, other donors)? 

4.3.  To  what  extent  has  the  sharing  of  roles  between  the  European  Commission  (DG 
ECFIN and other DGs), the IMF, Member States and others contributed to optimise the 
impact of the assistance? 

Some direct benefits of the EU intervention (Question 3.1) materialised in the form of 
implemented specific policy conditions  attached  to  the MFA operation in  question (as 
discussed  in  response  to  question  2.3).  This  has  mitigated  Georgia’s  external 
vulnerabilities  and  restored  market  confidence  (as  discussed  in  response  to  question 
2.1).  Other,  less  direct  benefits  came  about  from  supporting  structural  reforms  in 
Georgia more generally. Even if many of these reforms were already part of the national 
agenda, the MFA operation under evaluation helped to prioritise them and generate the 
necessary  political  will  for  their  implementation.  In  addition,  the  MFA  operation  has 
helped  to  mitigate  fiscal  pressures  (as  discussed  in  response  to  question  2.2)  and 
resulted in significant savings due to favourable terms of the assistance (as discussed in 
response to question 7).  

At  the  same  time,  the  visibility  of  the  MFA  operation  was  relatively  low,  with  many 
interviewed  stakeholders  not  being  able  to  distinguish  MFA  from  other  means  of  EU 

22 

 
 
support. The Commission will therefore consider ways of increasing the visibility of the 
MFA instrument.  

The additional  value of the MFA operation in  question,  in relation to assistance by 
other  donors,  mainly  resulted  from  efficient  and  complementary  sharing  of  roles 
between these programmes (Questions 3.1 and 3.2).  

As  an  example,  the  MFA  was  closely  linked  to  the  two  IMF  programmes  (the  SBA 
approved in 2014 and the EFF approved in 2017). MFA disbursements were part of the 
sources to cover the identified external financing gap. The close link was also evidenced 
in the deactivation of the MFA operation after the IMF SBA went off-track in 2015 and 
in a joint push to ensure the independence of the National Bank of Georgia. While there 
were no cross-conditions (identical policy measures), the Commission and the IMF built 
their  interventions  on  each  other.  For  instance,  the  MFA  policy  condition  on  capital 
adequacy  in  Georgian  banks  built  on  the  recommendations  of  the  Financial  Sector 
Assessment Programme (FSAP) carried out by the IMF in 2014. 

A similar sharing of roles was observable in relation to the World Bank. For example, 
the  MFA  policy  condition  on  strengthening  Georgia’s  public  procurement  system 
supported  the  World  Bank’s  technical  assistance  in  this  area.  Likewise,  the  MFA 
operation  in  question  (notably,  policy  measures  in  the  area  of  healthcare)  benefitted 
from the World Bank’s thorough engagement in the public healthcare sector (analytical 
Public Expenditure Reviews, work on accessibility of healthcare services etc.). 

The  MFA  operation  also  increased  the  effectiveness  of  bilateral  support  that  Georgia 
received from several EU Member States such as Germany and France. As this bilateral 
support  focused  on  infrastructure  development  in  the  energy  and  transport  sectors,  it 
was  enhanced  by  the  MFA  operation  that  targeted  institutional  progress  and,  in  this 
way, helped to create a more favourable environment for sectorial assistance. 

Evaluation Question 5: Coherence of the operation  

To what extent was the MFA operation in line with key principles, objectives and 
measures taken in other EU external actions towards Georgia? 

The  MFA  operation  of  2015-2017  was  part  of  a  comprehensive  package  of  the  EU’s 
assistance  to  Georgia.  The  EU  has  been  providing  substantial  (regular)  cooperation 
assistance to Georgia, in the form of budget support and technical assistance. In 2007-
2013, the EU provided  EUR 452 million  in  assistance to  Georgia under  the European 
Neighbourhood  Policy  Instrument.  The  indicative  allocation  of  regular  cooperation 
assistance for the period 2014-2020 is EUR 610 million to EUR 746 million under the 
European  Neighbourhood  Instrument.  Additionally,  the  European  Investment  Bank 
(EIB)  supported  projects  in  Georgia  with  a  total  loan  volume  of  more  than  EUR 
1.5 billion. 

As confirmed by stakeholder interviews and the Delphi survey, the EU’s interventions 
in  Georgia  were  closely  coordinated.  This  coordination  involved  the  relevant 
Commission  services  and  the  EU  Delegation  to  Georgia.  The  stakeholders  have  also 
confirmed  that  policy  conditions  of  the  MFA  operation  in  question  reflected  the 
Commission’s long-term agenda. 

In concrete terms, the MFA operation in question was aligned with the sectorial focus of 
the  EU’s  regular  cooperation  assistance.  For  instance,  in  the  area  of  public  finance 
management, the MFA policy condition on strengthening the independence of the State 

23 

 
 
Audit  Office  was  aligned  with  a  cooperation  assistance  operation  aimed  at  improving 
external audit. Likewise, a cooperation assistance operation for helping Georgia adapt to 
the requirements of the Deep and Comprehensive Free Trade Area (DCFTA) with the 
EU was supported by MFA policy conditions, all of which (except the one relating to 
risk management in banks) were linked to the requirements the EU-Georgia Association 
Agreement.29  Finally,  the  MFA  policy  condition  on  public  procurement  was  aligned 
with a cooperation assistance operation in the area of public administration reform. 

Both  the  EU’s  regular  cooperation  assistance  and  the  EIB  lending,  which  mostly 
supported  infrastructure  projects,  benefitted  from  macroeconomic  stability  and  more 
favourable institutional environment that MFA helped to ensure. In this way, the MFA 
operation under evaluation increased the effectiveness of other EU support operations in 
Georgia.  Importantly,  the  MFA  operation  of  2015-2017  complemented,  and  not 
replaced, other forms of the EU’s financial assistance to Georgia. 

Evaluation Question 6: Social impact of the operation  

What was the social impact of the MFA operation? 

6.1. What were the expected short and medium-term social effects of the assistance? 

6.2.  To  what  extent  have  the  expected  short  and  medium-term  social  effects  of  the 
assistance occurred as envisaged? 

6.3. What has been the contribution of the MFA to the occurrence of the expected social 
effects? 

Given  that  MFA  is  primarily  about  providing  an  exceptional  balance-of-payments 
support, the operation under evaluation did not directly pursue social objectives. At the 
same time, the MFA decision refers to beneficiary’s commitment to values shared with 
the EU such as  sustainable development  and poverty  reduction. Moreover, two policy 
conditions attached to the MFA operation in question (on the Healthcare Management 
and Utilisation Survey and the healthcare quality management unit) addressed the social 
safety net. 

In  terms  of  the  expected  social  effects  (Question  7.1),  the  evaluation  identified  the 
following expected short-term elements: (1) improvement in the quality and efficiency 
of the public healthcare services, and (2) smoothing the macroeconomic adjustment path 
and, in this way, helping to maintain social spending. The MFA operation in question 
did not specifically target medium-term social objectives but could have been expected 
to  help  reduce  Georgia’s  long-standing  social  problems,  notably  high  unemployment, 
poverty  and  inequality.  As  explained  in  Section  4,  some  limitations  due  to  data 
availability have been addressed by using other, qualitative evaluation methods. 

The degree to which these expected social effects materialised is not uniform (Question 
7.2). 

In order to assess changes in healthcare quality (one aspect of the first expected short-
term effect), the evaluation considers the neonatal (i.e. newborn) mortality rate, the life 
expectancy and the share of out-of-pocket  expenditure.30 While the neonatal mortality 
rate decreased significantly during the reference period (2014-2017), an increase in the 

29   DCFTA is part of this agreement. 
30   Spending paid by the by a patient directly to a healthcare provider, which may or may not be covered 

by an insurer or the state. 

24 

 
 
                                                           
life expectancy was more limited, as shown in Figures 15 and 16 respectively. The out-
of-pocket  expenditure  declined  both  as  share  of  total  spending  on  healthcare  and  as  a 
percentage of GDP, as shown in Figure 17. This mainly reflects the introduction of the 
Universal  Healthcare  System  in  2013,  which  increased  public  spending  and,  hence, 
availability of public healthcare services.  

Figure 15. Neonatal mortality (rate per 1,000) 

Figure 16. Life expectancy at birth (years) 

Figure 17. Out-of-pocket expenditure (% of health expenditure and GDP) 

25 

036912GeorgiaArmeniaUkraineMoldovaBelarusMortality rate, neonatal (per 1,000 live births)20142017656769717375GeorgiaArmeniaUkraineMoldovaBelarusLife expectancy at birth, total (years)201420164567894050607080902000200120022003200420052006200720082009201020112012201320142015Out-of-pocket expenditure (% of current health expenditure, LHS)Out-of-pocket expenditure (% of GDP, RHS) 
 
 
 
At the same time, changes in efficiency of public healthcare services (another aspect of 
the  first  expected  short-term  effect)  are  less  noticeable.  According  to  the  interviewed 
stakeholders, the capacity of the agency that manages the Universal Healthcare System 
remains limited. This, in turn, limits the agency’s ability to act as an active purchaser of 
healthcare  services  from  (mostly  private)  providers.  As  noted,  the  Commission 
acknowledges this issue that is, in fact, being addressed as part of policy conditionality 
of the current MFA operation approved in 2018. 

As  shown  in  Figure  18,  Georgia  maintained  an  elevated  level  of  social  spending  (the 
second expected short-term effect) during the reference period. Spending on healthcare 
was, in fact, one of the items that increased most. 

Figure 18. Georgia’s public spending (% of GDP) 

Assessing  the  medium-term  effects  is  more  complicated  because  they  might  have  not 
fully materialised yet, given the fact that the MFA operation was implemented relatively 
recently.  

The  level  of  unemployment  remained  elevated  during  the  reference  period  due  to  the 
skills  mismatch and low labour mobility  between the regions.  A  welcome decrease in 
youth  unemployment  was  largely  offset  by  an  increase  in  unemployment  among  the 
older  population.  The  Commission  is  aware  of  these  issues  and  is  helping  to  address 
them through policy conditionality of the current MFA operation.31 

31   Notably, the adoption of the Vocational Education and Training Law (which should help mitigate the 
skills  mismatch)  and  a  comprehensive  labour  market  strategy  (which  identifies  bottlenecks  and 
provides tools for addressing them, e.g. by establishing a public employment service). 

26 

0510152025303540051015202530354020042005200620072008200920102011201220132014201520162017Percent of GDPPercent of GDPExpendituresSocial protectionHealthEducationGeneral public servicesDefense, public safetyEconomic affairsOthersTotal  expenditure 
 
                                                           
Figure 19. Unemployment (% of active population) 

As  shown  in  Figure  20,  the  poverty  ratio  continued  to  decrease  during  the  reference 
period, albeit slower than in earlier years. The aforementioned reduction in the out-of-
pocket  spending  contributed  to  this  decrease.  Inequality  also  decreased  but  only 
marginally.  

Figure 20. Poverty (% of population) and inequality (Gini index, %) 

The  MFA  operation  of  2015-2017  contributed  to  the  aforementioned  social  effects 
both  directly  and  indirectly  (Question  7.3).  A  direct  contribution  came  in  the  form  of 
policy conditions that had a positive impact on the quality and efficiency of the public 
healthcare system, even if a number of problems remain. Indirectly, the MFA operation 
in  question  helped  to  ease  fiscal  pressures  and  sustain  social  spending.  Likewise,  the 
MFA  operation  contributed  to  restoring  macroeconomic  stability  that  supported 
economic growth and, hence, had a positive impact on employment, income and other 
social developments. 

It  is  difficult  to  fully  disentangle  the  social  effects  of  the  MFA  operation  under 
evaluation  from  other  factors  (such  as  support  operations  of  other  donors)32  and 

32   Similarly  to the MFA operation under evaluation, the IMF programmes (SBA of 2014 and EFF of 
2017)  supported  by  it  have  made  Georgia’s  adjustment  path  less  painful  and  helped  to  restore 

27 

051015202530350510152025303520042005200620072008200920102011201220132014201520162017Percent of avctive populationPercent  of active populationUnemployment (25 - 29 years old)Total unemploymentUnemployment (65+ years old)3034384246405060708020002001200220032004200520062007200820092010201120122013201420152016PercentPercent of populationPoverty headcount ratio, at $5.50 a day, 2011 PPP (LHS)GINI index (RHS) 
 
 
                                                           
quantify them. However, a positive social impact of the MFA operation of 2015-2017 
has  been  confirmed  by  the  stakeholder  interviews  and  the  Delphi  survey.  More 
specifically,  respondents  have  recognised  a  significant  role  that  the  MFA  operation 
played in maintaining social spending and encouraging reforms in related sectors such 
as education. 

Evaluation Question 7: Public Debt Sustainability of the operation  

What was the impact of the MFA operation on public debt sustainability? 

7.1  To  what  extent  has  the  MFA/IMF  assistance  contributed  to  returning  the  fiscal 
situation of Georgia to a sustainable path over the medium to longer-term? 

In order to assess the aforementioned impact, the evaluation considers the sustainability 
of  Georgia’s  public  debt  and  fiscal  savings  resulting  from  the  MFA  operation  in 
question.  

As  in  the  case  of  projections  of  external  debt,  the  evaluation  uses  the  IMF  DSA 
framework to quantify the effect on the sustainability of Georgia’s public debt. Three 
scenarios  have  been  computed  for  the  medium  (2014-2019)  and  long  (beyond  2019) 
term. The first scenario (A) is the baseline and incorporates the (actual) effects of both 
the  MFA  operation  of  2015-2017  and  the  IMF  SBA  approved  in  2014.  The  second 
(alternative)  scenario  (B)  assumes  that  neither  the  MFA  nor  the  IMF  SBA  were 
available to Georgia. Finally, the third (alternative) scenario (C) assumes that only the 
IMF SBA was available. 

As shown in Figure 21, the combined impact of the MFA and the IMF SBA (baseline, 
scenario  A)  helped  to  limit  the  increase  in  Georgia’s  public  debt.  A  similar,  although 
more limited, impact is observable in the alternative scenario C, assuming that only the 
IMF operation was available to  Georgia. This  latter, more limited impact is  explained 
by  a  smaller  size  of  the  MFA  operation,  compared  to  the  IMF  SBA.  The 
aforementioned effects are likewise observable in the long-term, meaning that the MFA 
operation  of  2015-2017  helped  to  return  Georgia’s  public  debt  to  a  sustainable  path. 
Analysis  shows  that  these  effects  materialised  mainly  through  the  confidence  channel 
(increased  confidence  reduced  the  risk  premium  on  Georgia’s  public  debt)  and  the 
economic growth channel (macroeconomic stabilisation supported growth). 

macroeconomic stability, both of which had positive social effects. These IMF programmes have not 
resulted in socially painful expenditure cuts or tax increases. 

28 

 
 
                                                                                                                                                                          
Figure 21. Projections of Georgia’s public debt-to-GDP ratio (%) 

As explained in response to Question 3.1, financial conditions of the MFA operation in 
question were very favourable. It is therefore appropriate to try to quantify the resulting 
fiscal savings. This is usually done by comparing concessional financing to regular debt 
issuances.  However, in  Georgia’s case, such comparison  is  not  straightforward, as the 
country  has  limited  market  financing  of  its  public  debt  (a  single  Eurobond  issued  in 
2011, with a 10-year maturity). 

In  order  to  estimate  the  net  present  value  of  the  MFA  grant  and  loan  components  in 
2015 (at the time of the first disbursement), the evaluation uses two discount rates: (1) 
5.25  per  cent,  which  was  the  average  remaining  yield  to  maturity  of  Georgia’s 
Eurobond at  that time, and (2) 7.35 per cent,  which is  a sum  of the first discount rate 
and the term premium of 210 basis points.33 As shown in Figure 22, the resulting fiscal 
savings of both grant and loan components are estimated at EUR 31.3-33.9 million, or 
0.26-0.28% of GDP. As noted, these savings (compared to market financing) helped to 
smooth  Georgia’s  adjustment  path  and  create  fiscal  space  for  reforms  and  sustained 
social spending.  

Figure 22. Fiscal savings due to MFA (EUR million and % of GDP) 

Assumed 

Estimated net present 

Estimated present 

"Savings" due to the 

"Savings" due to 

market 

value of the loan 

value of the grant 

favourable MFA 

favourable MFA 

interest rate 

component in 2015 (in 

component in 2015 

conditions (in EUR 

conditions (in GDP 

EUR million) 

(in EUR million) 

million) 

5.25% 

7.35% 

9.3 

12.27 

22.0 

21.7 

31.3 

33.9 

%) 

0.26% 

0.28% 

6.  CONCLUSIONS 

In  August  2013,  the  European  Parliament  and  the  Council  approved  a  Macro 
Financial  Assistance  operation  of  EUR  46 million  to  Georgia  (MFA-II).  This  was 
the  second  of  the  two  operations  pledged  by  the  EU  at  an  International  Donors’ 
Conference  in  Brussels  in  October  2008,  after  Georgia’s  armed  conflict  with  Russia. 

33   To reflect the fact that the  maturity of the MFA loan (15 years) was 5 years longer than that of the 
Eurobond (10 years). 210 basis points is the average term premium for 10-year maturities over 5-year 
maturities, for Georgia’s public borrowing in the national currency in 2012-2015. 

29 

253035404550552013201520172019202120232025202720292031203320352037203920412043204520472049Percent of GDPScenario AScenario BScenario C 
 
 
                                                           
Together  with  the  hit  from  the  global  financial  crisis,  the  conflict  with  Russia  put  the 
Georgian economy and its external financing under severe pressure, which necessitated 
international financial support. 

The  aim  of  this  ex-post  evaluation  of  MFA  is  to  assess  the  relevance,  efficiency, 
effectiveness, coherence, and EU value added of the MFA-II operation to Georgia, 
which was implemented in 2015-2017. It also explores the social impact of MFA and its 
effect on Georgia’s public debt sustainability and sets out to draw lessons. Limitations 
encountered  relate  predominantly  to  limited  data  coverage  in  some  areas  (social 
indicators), the changing economic environment over the extended period during which 
the  MFA-II  was  implemented,  and  the  fact  that  MFA-II  was  implemented  in  parallel 
with  other EU and international  support programmes (making it somewhat  difficult  to 
disentangle  the  impacts  of  the  MFA  operation).  Nevertheless,  these  do  not  put  into 
question the overall reliability of the evaluation analysis as they were mitigated by the 
fact  that  information  was  obtained  from  a  wide  range  of  sources,  using  different 
evaluation techniques, alternative scenarios and multiple rounds of feedback.   

The evaluation finds that the MFA-II was relevant in terms of its objectives, financial 
envelope  and  structural  conditions.  It  sought  to  alleviate  short-term  external  financing 
pressure, which was present both in 2011 (when the Commission submitted its proposal 
to  the  EU  co-legislators)  and  in  2014  (when  the  MoU  was  negotiated  and  the 
implementation started). The overall design of the operation (subjecting only the second 
instalment to policy conditionality) is also judged as appropriate. This design allowed a 
swift first disbursement, to provide short-term relief, while also encouraging structural 
reforms by means of policy conditionality of the second disbursement. 

The areas of MFA policy conditionality covered the most relevant reform challenges in 
Georgia and were  relevant for the macroeconomic stabilisation. The structural  effects 
of  the  operation  largely  materialised  as  planned,  as  Georgia  implemented  all  relevant 
policy conditions. Importantly, evidence gathered shows that the relevant reforms were 
implemented faster than would have been the case without the MFA operation. There is 
an argument that some additional policy measures in healthcare and banking could also 
have  been  relevant,  however  this  has  to  be  balanced  with  the  need  to  keep  the  list  of 
MFA conditions focused. Some of these measures have subsequently been included as 
policy conditions in the current MFA operation to Georgia approved in 2018. 

to 

The amount of MFA-II was appropriate (in terms of burden sharing with other donors) 
and  proportional  (limited 
to  achieve  short-term 
macroeconomic  stability  in  Georgia).  Moreover,  the  evaluation  finds  that  the  amount 
was relevant at the time of the actual disbursements, helping to close the financing gap 
identified  as  part  of  Georgia’s  programmes  with  the  IMF.  The  financial  terms  of  the 
MFA  operation  (half  of  the  amount  was  provided  in  grants  and  another  half  in  loans) 
was also appropriate. 

the  minimum  necessary 

The  evaluation  finds  that  the  MFA-II  was  effective  in  helping  to  improve  Georgia’s 
balance-of-payments situation, as well as supporting fiscal consolidation and structural 
reforms.  Indeed,  together  with  the  IMF  arrangements,  the  MFA  operation  helped  to 
limit  the  increase  in  Georgia’s  external  debt  and  increase  confidence  in  the  Georgian 
economy (by decreasing the market-based financing cost and dollarisation, as well as by 
helping to stabilise the national currency). 

The MFA-II was designed and implemented efficiently, and in close coordination with 
the  Georgian’s  authorities,  the  IMF  and  the  World  Bank.  Delay  in  the  adoption  was 
caused  by  procedural  disagreements  between  the  co-legislators.  However,  the 
evaluation finds that this exceptionally long timeline of approving the operation did not 

30 

 
result in efficiency losses, because from late 2010 until July 2014 and from mid-2015 
until April 2017, Georgia did not have a disbursing IMF programme  (a precondition for 
granting  MFA).  The  latter  would  have  prevented  disbursements  of  MFA  even  if  the 
operation  under  evaluation  had  been  approved  earlier.  Moreover,  the  Commission 
asserts that the timeframe between when the Georgian authorities submitted the official 
request for MFA request (May 2010) until the time when the proposal was submitted to 
the EU co-legislators  (January 2011) is  in  effect  within normality,  as  the  Commission 
could only start preparing a new proposal once the previous MFA-I operation was fully 
disbursed  and  complete  (August  2010).  At  the  same  time,  the  Commission  will 
endeavour  to  shorten  this  period  whenever  possible,  as  has  already  been  the  case  in 
several  subsequent  proposals.  In  the  subsequent  MFA  programme  to  Georgia  34  for 
instance, the Commission adopted the proposal for further MFA roughly three months 
after it received the request from the Georgian authorities. The issue of MFA timelines 
will  be  further  examined  in  a  wider  MFA  meta-evaluation  (elaborated  in  Annex  1) 
planned to commence in 2020.  

The MFA operation under evaluation was a coherent part of a comprehensive package 
of the EU’s assistance to Georgia. Indeed, the evaluation confirms  close coordination 
between  MFA-II  and  other  EU's  interventions.  The  EU’s  added  value  was  most 
apparent in stimulating the structural reform process in Georgia. In addition, the MFA 
operation helped to  mitigate fiscal  pressures and resulted in  significant  savings due to 
favourable terms of the assistance. Despite these benefits, the evaluation finds that the 
visibility  of  MFA-II  was  relatively  low.  The  issue  of  MFA  operational  visibility  will 
also be further examined in a MFA meta-evaluation planned to commence in 2020.  

The MFA-II had an overall positive social impact. It helped to improve the quality and 
efficiency of the public healthcare services, although progress in this area is uneven and 
is  being  addressed  as  part  of  policy  conditionality  of  the  current  MFA  operation 
approved  in  2018.  The  MFA  operation  also  helped  to  smooth  the  macroeconomic 
adjustment  path  and,  in  this  way,  maintain  social  spending.  More  broadly,  the 
evaluation  asserts  that  the  MFA  operation  contributed  to  restoring  macroeconomic 
stability  that  supported  economic  growth  and,  hence,  had  a  positive  impact  on 
employment, income and other social developments.  

Finally, the evaluation finds that the MFA-II had a positive effect on the sustainability 
of  Georgia’s  public  debt  and  enabled  fiscal  savings.  Together  with  the  IMF 
arrangements,  the  MFA-II  helped  to  limit  the  increase  in  Georgia’s  public  debt.  The 
fiscal savings of around 0.3% GDP resulted from very favourable financial conditions 
of the MFA operation. These savings (compared to market financing) helped to smooth 
Georgia’s  adjustment  path  and  create  fiscal  space  for  reforms  and  sustained  social 
spending. 

34 Decision (EU) 2018/598 (OJ L 103, 23.4.2018, p. 8–13) 

31 

 
 
                                                           
Annex 1: Procedural information 

Organisation, design and timing 

This  evaluation  assesses  the  MFA  operation  in  Georgia,  in  the  amount  of  EUR 
46 million,  which  was  approved  in  2013  and  implemented  in  2015-2017.  The 
assessment is in line with the Financial Regulation (Article 30(4)) and the relevant MFA 
Decision,  which  require  the  European  Commission  to  submit  an  ex-post  evaluation 
report to the European Parliament and the Council.  

The  objective  of  the  evaluation  is  to  draw  lessons  with  respect  to  the  EU’s  financial 
assistance, in particular the design and implementation of the programmes and the way 
they  contributed  to  achieving  macroeconomic  stabilisation  and  fostering  structural 
reforms.  Apart  from  identifying  areas  of  improvement  for  similar  on-going  or  future 
possible  interventions,  the  evaluation  also  aims  at  ensuring  better  transparency  and 
accountability of the Commission’s activities.  

The  evaluation  looks  at  various  aspects  of  this  particular  EU  intervention  (relevance, 
effectiveness,  efficiency,  EU  added-value,  coherence  with  other  EU  policies  towards 
Georgia, as well as the social impact and the impact on the sustainability of Georgia’s 
public debt). In order to ensure validity, the analysis and conclusions are based on the 
evidence  obtained  using  several  evaluation  methods  (desk  research,  stakeholder 
interviews, focus groups and a Delphi survey). 

DG  ECFIN,  which  was  the  lead  DG,  chaired  the  ISG  that  was  set  up  to  manage  the 
evaluation.  Apart  from  DG  ECFIN,  the  ISG  comprised  of  representatives  of  other 
Commission  services  (the  Secretariat  General  and  the  Directorate-General  for 
Neighbourhood and Enlargement Negotiations) and the EEAS. 

DG ECFIN hired an external contractor to complete an evaluation report, which informs 
this SWD. A kick-off meeting, where the ISG and the external contractor discussed the 
deliverables  and  the  evaluation  methods,  took  place  in  September  2018.  This  was 
followed by meetings on the inception and interim reports in, respectively, October and 
December 2018. The final report was submitted in April 2018. In addition to meetings, 
ISG  members  were  continuously  informed  and  consulted  (via  email  and  by  phone) 
during the evaluation. 

The work of the external contractor was complemented by internal analysis. 

Consultation of stakeholders 

The objective of stakeholder consultation was to collect as much valuable and relevant 
economics-based  inputs  as  possible  from  various  groups  and  people  involved  to 
construct an ex-post assessment of the design, implementation and impact of the MFA-
II  operation. This  did  not  include an open public consultation,  but  was rather targeted 
towards specialists - people with an informed economic understanding of the Georgian 
MFA-II operation and the context in which it was implemented either by being closely 
involved its development and/or the implementation or persons with expert knowledge 
in  the  areas  related  to  the  objectives  of  the  operation.  In  addition,  an  indicative 
evaluation  roadmap  was  published  in  June  201835  to  seek  wider  feedback.  Finally,  a 
number  of  the  conclusions  and  limitations  identified  will  require  further  consideration 
and may benefit from cross-comparison with other recently completed MFA operations. 
A meta-evaluation  is  planned for 2020  and it will explore these points in  more detail. 

35   Available at: https://ec.europa.eu/info/law/better-regulation/initiatives/ares-2018-3053917_en. 

32 

 
 
                                                           
The meta-evaluation will provide a synthesis of the results of the evaluation studies of 
MFA  operations  carried  out  in  the  last  ten  years  and  will  consider  the  reliability  and 
relevance of the methodology, process, outputs, and outcomes of these evaluations. In 
addition the exercise will assess the timeline of MFA operations and potential linkages 
to  added-value  and  operational  visibility.  An  open  public  consultation  will  be 
undertaken as part of the meta-evaluation to gather the views of wider stakeholders. 

33 

 
 
 
Annex 2: Stakeholder consultation strategy, including summary of 
results 

This  stakeholder  consultation  strategy  was  developed  with  the  overall  objective  to 
capture as much valuable and relevant information as possible with regard to the MFA-
II operation in addition to information collected through review of key documentation 
and  communication,  consultation  of  EC  officials,  and  data  analysis.  The  consultation 
focused  on  extracting  recollections  from  the  time  period  in  which  the  operation  was 
designed  (2011–2014)  and  implemented  (2015–April  2017),  but  also  on  collecting 
views on the period after the MFA-II was ended (May 2017– 2018) to assess its impact 
and sustainability.  

This consultation strategy: 
  sets out the objectives of the consultation;  
  maps key stakeholders;  
  presents the consultation methods and tools which are used; and  
  demonstrates how the stakeholder consultation fits in the evaluation framework.  

Table A.1.1 presents  a detailed timeframe for the implementation  of this consultation. 
The items listed in this timeframe are elaborated in the following sections. 

34 

 
 
 
 
 
Table A.1.1 Time schedule of the implementation of the stakeholder consultation  

OCTOBER 2018 

Brussels 

9; 17/19 October 

CONSULTATION TIME SCHEDULE 

NOVEMBER 2018 

JANUARY 2019 

Home based 

20 November 

Tbilisi, Georgia 

12-16 November 

Home-based 

22-27 November  

Tbilisi, Georgia 

15-18 January  

Home-based 

Week of January 21 

Georgia 

authorities 

Semi-

structured 

interviews 

IFIs 

IFIs, Other 

donors 

Georgia 

authorities 

Other 

donors 

Semi-structured 

interviews 

IFIs 

EC 

Semi-

IMF 

Semi-

External 

Focus group 

Georgia 

representative

structured 

structured 

experts 

structural and 

authorities 

s 

interviews 

interviews 

social reforms 

Delphi survey 

(online survey + 

follow up) 

External 

Focus group 

Georgia 

experts  

macroeconomic 

authorities 

and fiscal reforms 

Delphi survey 

(online survey + 

follow up) 

35 

 
 
 
 
 
 
 
 
 
 
Objective setting 

The objective of stakeholder consultation was to collect as much valuable and relevant 
information as possible from various groups and people involved to construct an ex-post 
assessment  of  the  design,  implementation  and  impact  of  the  MFA-II  operation.  The 
external evaluators consulted stakeholders on the following key aspects: 
  Relevance  of  the  MFA-II  operation:  the  relevance  of  the  objectives,  the  financial 
envelop  and the conditionality, both  at  the time  of designing  the MFA-II operation 
(2011-2014) and with the benefit of hindsight; 

  Effectiveness  of  the  MFA-II  operation:  the  direct  results  of  the  operation,  i.e.  its 

results on macroeconomic level and in the area of the structural reforms; 

  Efficiency of the MFA-II operation: the design and process of the MFA-II in terms 

of value-for-money; 

  EU-added  value  of  the  MFA-II  operation:  the  added  value  of  the  operation  when 

considering other possible scenarios and alternatives; 

  Coherence  of  the  MFA-II  operation:  alignment  with  other  support  initiatives 

implemented at the time of the MFA-II; 

  Social  impact:  the  more  indirect  impact  of  the  MFA-II  operation  in  the  context  of 

social development in Georgia; 

  Debt sustainability: the longer-term result of the MFA-II in terms of implications to 
Georgia’s  government  and  external  debt  dynamics  and  the  fiscal  and  external 
sustainability.  

Consultation was thus partly related to recalling past events, but also to collect current 
opinions, which can be made with the benefit of hindsight. The aim was thus to gain an 
understanding of the decision-making at the time of the design and implementation of 
the MFA operation, but also to identify the actual relevance and impact of the operation.  

While  stakeholders  were  asked  to  make  (subjective)  assessments  and  express  their 
personal opinions, they were encouraged to refer to written sources wherever possible. 
Eventually, the results of the stakeholder consultation were triangulated  with  data and 
documentation  to  provide  well-evidenced  responses  to  Evaluation  Questions  (as 
demonstrated in the last section of this annex). 

Stakeholder mapping 

Consultation  was  targeted  to  specialists  –  either  people  who  have  either  been  closely 
involved in the development and/or the implementation of the MFA operation or people 
with expert knowledge in the areas related to the objectives of the MFA operation (i.e. 
macroeconomic  and  fiscal  policy,  and  structural  reforms  in  the  areas  of  PFM,  social 
policy, financial sector, trade and competition policy).  

Below  are  the  four  groups  of  stakeholders  that  had  a  central  role  in  this  consultation 
strategy: 

Georgia public Institutions 

1. 
Obviously, the recipient was an important stakeholder to consult on the key aspects, in 
order to incorporate the beneficiary’s view on the MFA2 operation.  

36 

 
 
 
 
 
 
 
After having gone through all documentation provided and collected on the MFA-II, the 
following  key  institutions  within  the  Government  of  Georgia  were  invited  for  a 
discussion  on  the  design  and  the  implementation  MFA-II  operation,  and  its 
macroeconomic and fiscal effects:  
  Ministry of Finance (MOF): implementing Ministry of the MFA loan; 
  National Bank of Georgia (NBG): implementing financial institution; 
  Parliamentary Budget Office (PBO). 

Within  these  organisations,  a  distinction  should  be  made  between  high-level  policy 
makers  and  the  technical  staff  working  on  actual  implementation.  The  latter  group  is 
able  to  specifically  comment  on  the  efficiency  of  the  implementation  of  the  MFA-II 
operation. 

Furthermore, a number of other key stakeholders within the government were consulted, 
specifically on the relevance, effectiveness, and impact of the conditions for structural 
reforms: 
  State Audit Office (SAO): on PFM reforms (action 1 and 2); 
  State Procurement Agency on action 1; 
  Ministry of Labour, Health and Social Affairs on social reforms (action 3 and 4); 
  National Bank of Georgia (NBG) on financial sector reforms (action 5 and 6); 
  Ministry of Economy and Sustainable Development on trade and competition policy 

(action 7 and 8). 

International Financial Institutions 

2. 
As major international financial institutions (IFIs), the IMF and World Bank were key 
stakeholders  for  consultation  as  they  were  to  some  extent  involved  in  the  preparation 
and  implementation  of  the  MFA-II.  In  addition,  they  provided  similar  support  to 
Georgia  (the  IMF  via  a  SBA  and  an  EFF,  the  World  Bank  via  DPLs).  Both 
organisations  could  thus  provide  input  to  all  key  aspects  of  the  evaluation.  They  are 
probably  not  the  group  with  the  largest  interest  in  this  MFA  evaluation,  but  they  do 
have a significant influence. 

Georgia external (i.e. non-governmental) experts 

3. 
There is a variety of actors, who were not directly involved in the MFA operation itself, 
but  are  very  knowledgeable  on  the  topic  of  macroeconomic  and  fiscal  developments, 
and on structural and social reforms in Georgia. It was important to consult these actors 
as well in order to determine the actual relevance, effectiveness and impact of the MFA-
II, as they possess the knowledge to place the MFA-II in the wider context of Georgia’s 
economic and social situation. These actors include:  
  Ex-government  officials  and  (ex-)  Parliamentarians:  To  give,  wherever  possible,  a 

more external view on the structural conditions of MFA; 

  NGO,  academics  and  other  interest  groups:  To  provide  an  outsider’s  view  on  the 
economic and social developments that have taken place in Georgia in the period of 
2011-2018 and on the structural reforms which have taken place in Georgia since the 
MFA; 

  Banks  and  financial  institutions:  Officials  from  private  banks  in  Georgia  could 
provide an external view on the economic and financial developments in Georgia and 
on the current macroeconomic and fiscal situation.  

4. 

Other donors 

37 

 
 
 
 
 
 
A selection of other donors was consulted, to provide a more outsider’s’ opinion on the 
MFA operation in the context of wider aid provisions and to gain further insight in the 
coherence of the MFA operation. They  do not  have a strong interest  or influence, but 
are  interesting  to  get  a  better  understanding  of  the  context  in  which  the  MFA  was 
provided, and possibly can put the relevance and impact into context. These include:  
  US Agency for International Development; 
  German Development Agency, GIZ; 
  World Health Organisation. 

Two  focus  group  sessions  were  organized  with  a  distinct  focus.  The  first  session 
covered  structural  and  social  reforms  in  Georgia,  and  focused  on  the  relevance  of  the 
MFA  conditions.  For 
invitees  consisted  of  (former) 
Parliamentarians, academics and non-governmental organisations.  

this  focus  group, 

the 

Representatives  of  the  following  organisations  participated  in  the  first  focus  group 
discussion: 

 
Ilia State University 
  Tbilisi State University 
  Transparency International - Georgia  
  The Association of Young Financiers and Businessmen; 
  The Georgian Economics  
  Parliamentary Committee on Budget and Finance  
  Georgian Taxpayers Union 
 
  PMC Research  
  European Bank for Reconstruction and Development. 

ISET  

The  second  focus  group  session  focused  on 
the  macroeconomic  and  fiscal 
developments,  including  topics  like  Georgia’s  financing  needs,  debt  sustainability  as 
well as the financial sector reforms. Participants were experts from the financial sector. 
The focus was on senior economists residing in  special economic research or analysis 
departments  of  the  banks  and  other  financial  institutions.  (See  the  detailed  list  of  the 
participants below). 

Representatives  of  the  following  organisations  participated  in  the  first  focus  group 
discussion: 

Investment Arm of Bank of Georgia 

  Procredit Bank Georgia 
 
  Georgia Capital PLC  
  Basis Bank 
  FINCA Bank - Georgia,  
  VTB Bank 
  MFO Crystal  
  Terabank  
  Pasha Bank 
  TBC Capital. 

The full list of participants is detailed in Annex 1 of the external report.  

38 

 
 
 
 
 
Consultation methods 

Related  to  the  four  groups  of  key  stakeholders  above,  the  evaluators  used  a  targeted 
consultation approach, utilising three key tools: 

Semi-structured interviews 

1. 
The objective of the interviews was to extract detailed information on the following:  
  MFA design and implementation; 
  results of MFA on the macroeconomic and fiscal situation; 
  results in the fields of the structural reform conditions; 
  social impact; 
  debt sustainability.  

Interviews  were  thus  carried  out  particularly  with  the  first  two  stakeholders,  who  are 
well  aware  of  the  MFA  instrument  and  its  implementation.  Interviews  were  also 
conducted  with  the  fourth  group  (other  donors),  but  these  interviews  were  focused  on 
the coherence of MFA with other donor initiatives and were less in-depth.  

The format of semi-structured interviews was chosen on purpose: on the one hand, this 
format offers the possibility to discuss a few set topics with the interviewees. Details 
were asked on events which happened in the past, therefore pre-interview questionnaires 
were sent out. These questionnaires contained a brief overview of key bullet points that 
the evaluators would like to discuss, to enable the interviewee to prepare him/herself by 
collecting information in advance. On the other hand, semi-structured interviews leave 
room for the interviewer to raise other relevant issues, also in feedback to answers of the 
interviewee.  

The most important findings of the semi-structured interviews are listed in the following 
table: 

No.  Most important findings of the semi-structured interviews 
1. 
2. 

The blend of grant and medium-term concessional loan of the assistance was appropriate. 
The most important reform areas were covered by the conditions and the actions set by the 
MoU were all relevant or highly relevant. 
Additional measures could have targeted (i) ensuring the conditions for SAO’s revenue 
auditing; (ii) improvement of internal auditing; (iii) improving the human and infrastructural 
capacities of Social Services Agency (iv) supporting responsible lending practices; (v) 
support filling the knowledge gap of companies related to the DFCTA; and (vi) improving 
the human and infrastructural capacities of the Competition Agency. 
Public procurement needs to be further developed. Lack of sufficient progress in the area of 
the health care (cost effectiveness) and the competition policy was mainly due to the lack of 
sufficient human and infrastructural capacities. Implementation of the DFCTA is very costly 
and the beneficial impact on trade performance is not yet tangible.  
The primary added value of the operation is its important role in promoting structural 
reforms. Getting an external credibility stamp on policy reforms is a highly important factor 
in Georgia.  

3. 

4.  

5. 

6.  MFA-II also had a prominent role in helping to alleviate external pressure, primarily by 

supporting the replenishment of FX reserves and in helping to restore market confidence.  

39 

 
 
 
 
 
7. 

8. 

9. 

The short-term impact on budget financing was a less important contribution of the MFA 
operation. 
The conditions were coherent with and well complemented the operations of other IFIs, 
which was well appreciated by non-IFI stakeholders as well. There was an efficient division 
of labour among IFIs.  
The pace and the ambition of the fiscal consolidation plan set by the IMF was reasonable. 
Deficit targets were missed due to the lack of the government’s commitment.  

Expert focus group 

2. 
Focus groups aim to gain information of the MFA in a wider context: what has been its 
relevance and its impact if the operation is put in a broader perspective.  

Focus  groups  are  ideal  for  exploring  people's  experiences,  opinions,  wishes  and 
concerns and have been identified as especially useful for studying the success or failure 
of  particular  policies  and  programmes.  Specifically  for  this  MFA,  organising  these 
focus groups helped in understanding the current paradigm of reforms and get a better 
understanding on how MFA has been tailored to the local situation in Georgia. 

This  instrument  was  particularly  useful  for  the  third  stakeholder  group:  these  experts 
have been or are too far away from the MFA operation to conduct detailed one-on-one 
interviews,  but  their  participation  in  a  group  discussion  would  be  very  useful  to  gain 
deeper understanding of the macro/fiscal developments and the structural/social reforms 
in  Georgia.  Discussion  were  restricted  to  two  hours,  to  encourage  participation  of 
people  and  ensured  participants  that  Chatham  House  rules  were  to  be  applied.  These 
rules  elicit  the  maximum  amount  of  input  from  the  participants  and  therefore  provide 
the best opportunity to contribute to addressing the evaluation questions with valuable 
stakeholder insights that may not be possible in a more open forum.  

Two group sessions were organised, each with a specific focus: 

a. 

b. 

Structural and social reforms in Georgia  
  Relevance of the MFA conditions in light of Georgia’s developments; 
  Reforms in: PFM, social safety net, trade and competition policy; 
  Development  of  social  indicators  regarding  employment  and  poverty  and 

inequality. 

Macroeconomic, fiscal and financial sector developments in Georgia 
  Georgia’s financing requirements; 
  Internal and external factors of Georgia’s external financial situation; 
  Pace, ambition and composition of appropriate financial consolidation; 
  Debt sustainability; 
  Financial sector reforms. 

The first focus group took place during the first mission to Georgia in November 2018. 
For this  focus  group, invitees  included (former)  Parliamentarians, academics and non-
governmental  organisations  (see  list  of  participants  above).  The  second  focus  group 
took place during the second mission in January 2019 and involved officials from banks 
and financial institutions (see list of participants above).  

The most important findings of the Focus Group discussions are listed in the following 
table: 

40 

 
 
 
 
 
 
 
 
No.  Most important findings of the Focus Group discussions 
First Focus Group on structural and social reforms 
1.  Most of the important reform areas were covered by the conditions.  
2. 
3. 

Conditions et in the MoU were relevant. 
One of the most relevant action was the one supporting the independence of SAO. 
Independence of SAO strengthened and the office produces great reports, but the 
recommendations are not always implemented due to the lack of political will.  
Low cost effectiveness of the UHC is a major problem of the health care system.  
Level of social expenditure is high, there is a need to improve its efficiency. 
High unemployment and skill mismatches are lingering problems of the Georgian 
economy. 

4. 
5. 
6. 

Second Focus Group on macroeconomic, fiscal and financial sector developments 
1. 

The IMF/EU programme caused a 110 basis points decrease in the country risk 
premium. 
Group discussions gave important inputs for the long term assumptions to the debts 
sustainability analysis. 
ICAAP condition (Action 5) was highly relevant both in terms of the structural 
reform process and the economic challenges. 
Additional measures could have support the introduction of responsible lending 
practices. Lack of appropriate regulation in the past and over indebtedness of low 
income households is a pressing problem for the economy.  
External debt sustainability, high dollarization (both in the public and private balance 
sheets) are still relevant problems of the economy. 

2. 

3. 

4. 

5. 

Delphi method 

3. 
The Delphi method is an evaluation methodology that relies on judgmental estimates of 
experts  based  on  their  insights  and  collective  knowledge.  The  external  evaluators 
applied  a  light  version  of  the  Delphi  method  as  an  additional  consultation  tool.  The 
objective of using this tool was to gain further insight into the added value of the MFA 
operation.  In  November,  after  the  first  field  mission,  the  evaluators  identified  a  few 
possible scenarios related to the Georgia’s macroeconomic and fiscal developments and 
structural  reforms.  The  main  question  to  the  participants  in  the  Delphi  survey  was  to 
what  extent  the  MFA  operation  has  contributed  to  certain  macroeconomic  and  fiscal 
developments,  and  in  the  field  of  structural  reform,  by  considering  what  would  have 
happened if the MFA would not have been granted. 

In  operational  terms,  the  Delphi  method  involved  two  rounds  of  consultations.  The 
external  evaluators  sent  the  panel  members  a  link  to  an  electronic  questionnaire 
powered by the CheckMarket Survey Tool. They also provide them the opportunity to 
re-assess their position. The panel included 16 experts. It is a balanced mix of Georgian 
officials,  EU  officials,  either  based  in  Brussels  or  in-country,  and  representatives  of 
other IFIs. This allowed us to receive a sufficient amount of valid responses. It should 
be  emphasised  that  the  quality  of  result  is  not  so  much  related  to  the  number  of 
respondents,  but  rather  to  the  level  of  knowledge  of  the  expert  panel.  Regarding  the 
survey  instrument,  the  experience  with  previous  MFA  evaluations  strongly  suggested 
the  use  of  a  simplified  questionnaire,  focusing  on  a  limited  set  of  key  variables.  This 
increases the response rate and favour the emergence of a consensus opinion.  

The most important findings of the Delphi survey are listed in the following table: 

41 

 
 
 
 
 
No.  Most important findings of the Delphi survey 
1. 

The impact of MFA has been primarily through the promotion of structural reforms and 
the easing the balance-of-payment pressure in Georgia. 
The general design of the operation (i.e. with the first instalment linked to the IMF SBA 
progress in general and the second instalment linked to the structural conditions) was 
optimal. 
The design of the conditionality was appropriate. The MoU included a balanced mix of 
conditions, the complexity and the number of the actions were proper. 
The conditions set by the MoU were important or fairly important. 

4. 
5.  All conditions were completed satisfactorily.  
6. 

The main value added of the conditionality was through the speeding up the 
implementation of the reforms. 
The alignment of the conditions with other support programmes was appropriate.  
The MFA operation made a significant difference in enabling Georgia to sustain the costs 
for social provisions. The non-social related structural conditions had a sizeable indirect 
positive impact on social developments. 
The social safety net conditions attached to the MFA helped to reinforce or kick-start 
some reforms in the social sector. 

2. 

3. 

7. 
8. 

9. 

The consultation strategy and the evaluation framework 

Table  A.1.4  combines  the  different  groups  of  stakeholders  and  the  consultation 
methods, and shows how they contributed to answering the evaluation questions.  The 
table  also  mentions  other  sources,  which  provided  information  that  could  be 
triangulated with the information collected from stakeholder consultation.  

42 

 
 
 
 
 
 
 
Table A.1.4 Stakeholder consultation in the evaluation framework 
EQ criteria 
RELEVANCE 

Sub-criteria 
Relevance of 
objectives 

Stakeholders 
Georgia Authorities: MOF, NBG 
IFIs 
Other donors 
Georgia authorities, IFIs 
External experts (officials from banks 
and financial institutions) 

Relevance of the 
financial 
envelope 

Georgia Authorities: MOF, NBG 
IFIs  
External experts (officials from banks 
and financial institutions) 

Relevance of 
conditions 

EFFECTIVENESS  Effectiveness in 

terms of 
macroeconomic 
stabilisation 

Georgia Authorities: selected 
organisations/ ministries for structural 
reforms 
IFIs 
External experts (Parliamentarians and 
NGO/academics) 
Georgia Authorities: MOF, NBG 
IFIs 
External experts (officials from banks 
and financial institutions) 
External experts (officials from banks 
and financial institutions) 

Effectiveness in 
terms of fiscal 
policy 

Georgia Authorities: MOF, NBG 
IFIs 
External experts (officials from banks 
and financial institutions) 

43 

Consultation method 

Semi-structured interviews 

Delphi survey 
Focus group on 
macroeconomic and 
fiscal/financial topics 
Semi-structured interviews 
Delphi survey 
Focus group on 
macroeconomic and 
fiscal/financial topics 

Semi-structured interviews 
Delphi survey 

Focus group on structural 
reforms 

Semi-structured interviews 

Focus group on 
macroeconomic and 
fiscal/financial topics 

Semi-structured interviews 

Focus group on 
macroeconomic and 
fiscal/financial topics 

Other sources 
  Consultation of EC staff; 
  Study of documentation. 

  Consultation of EC staff; 
  Economic/financial 
background papers. 

  Consultation of EC staff; 
  Case studies on financial 
sector and trade policy; 
  Reports on socio-economic 

situation in Georgia. 

  Consultation of EC staff; 
  Debt sustainability analysis; 
  Data analysis; 
  Document review of 

macroeconomic reports. 

  Consultation of EC staff; 
  Descriptive quantitative 

analysis; 

  Document review of fiscal 

reports. 

 
EQ criteria 

Sub-criteria 
Effectiveness in 
terms of 
structural 
reforms 

EFFICIENCY 

NA 

Stakeholders 
Georgia Authorities: selected 
organisations/ 
ministries for structural reforms 
IFIs 
External experts (Parliamentarians and 
NGO/academics) 
Georgia Authorities: MOF, NBG (more 
technical staff) 

Consultation method 
Semi-structured interviews 

Semi-structured interviews 
Focus group on structural 
reforms 

 Semi-structured interviews 

EU ADDED 
VALUE 

NA 

Georgia Authorities: MOF, NBG 
IFIs 

Semi-structured interviews 

  Delphi survey 

Other sources 
  Consultation of EC staff; 
  Data analysis; 
  Document review of Georgia 

reforms; 

  Case studies on financial 
sector and trade policy. 
  Consultation of EC staff; 
  Desk research, review of 

documentation/communicatio
n. 

  Delphi survey to key EC staff; 
  Documentation review and 
interviews with EC staff on 
the rationale. 

COHERENCE 

NA 

Georgia Authorities: MOF, NBG 
Other donors 

Semi-structured interviews 
Semi-structured interviews 

  Consultation of EC staff; 
  Review of programme 

SOCIAL IMPACT  NA 

DEBT 
SUSTAINABILITY 

NA 

Georgia Authorities: selected 
organisations/ ministries for structural 
reforms 
IFIs 
Other donors 
External experts (Parliamentarians and 
NGO/academics) 
Georgia Authorities: MOF, NBG 
IFIs 
External experts (officials from banks 
and financial institutions) 

Semi-structured interviews; 
Delphi survey 

Focus group on structural 
reforms 
Semi-structured interviews 

Focus group on 
macroeconomic and fiscal 
topics 

44 

documentation and EU / other 
donor programmes; 

  Case studies. 
  Consultation of EC staff; 
  Document and data analysis of 

social variables. 

  Consultation of EC staff; 
  Debt sustainability analysis; 
  Document review of fiscal 

reports. 

 
45 

 
 
List of key MFA-II documents reviewed  

European  Commission  DG  ECFIN  (2010).  Proposal  for  Further  Macro-Financial 
Assistance  to  Georgia  for  2010-11,  Note  for  the  Economic  and  Financial  Committee. 
ECFIN/D2/AJ/ic Ares(2010)SN680365. 

European  Commission  DG  ECFIN  (2011a).  Proposal  for  a  Decision  of  the  European 
Parliament  and  of  the  Council  providing  further  macro-financial  assistance  to  Georgia. 
13.1.2011, COM(2010) 804 final. 

European  Commission  DG  ECFIN  (2011b).  Ex-ante  evaluation  statement  on  further 
macro-financial  assistance  to  Georgia,  Accompanying  document  to  the  Proposal  for  a 
Decision  of  the  European  Parliament  and  of  the  Council  providing  further  macro-
financial assistance to Georgia. 13.1.2011, SEC(2010) 1617 final. 

Decisions No 778/2013/EU of the European Parliament and of the Council of 12 August 
2013  providing  further  macro-financial  assistance  to  Georgia,  Joint  Declaration  by  the 
European  Parliament  and  the  Council  adopted  together  with  the  decision  providing 
further  macro-financial  assistance  to  Georgia,  Official  Journal  of  the  European  Union, 
14.8.2013 L218/15-23. 

European Commission DG ECFIN (2014a). Report on mission to Georgia: Memorandum 
of Understanding negotiations for Macro-Financial Assistance to Georgia, (Tbilisi, 10-14 
June 2014). Ref. Ares(2014)2331983 - 14/07/2014. 

European  Commission  DG  ECFIN  (2014b).  Report  on  mission 
to  Georgia: 
Memorandum of Understanding negotiations for Macro-Financial Assistance to Georgia, 
(Tbilisi, 28 July -1 August 2014). Ref. Ares(2014)2620777 - 07/08/2014. 

Association  Agreement  between the European Union  and the European  Atomic Energy 
Community  and  their  Member  States,  of  the  one  part,  and  Georgia,  of  the  other  part, 
Official Journal of the European Union, 30.8.2014 L261/4-743. 

Memorandum of Understanding between the European Union as Donor and Georgia as 
Beneficiary  and  the  National  Bank  of  Georgia  as  Beneficiary's  Financial  Agent, 
11.12.2014. 

Macro-Financial Assistance for Georgia Loan Facility Agreement, 11.12.2014. 

Macro-Financial Assistance for Georgia Grant Agreement, 17.12.2014. 

European Commission DG ECFIN (2015a).European Union Macro-Financial Assistance: 
Disbursement  of  the  1st  Instalment  of  EUR  10,000,000  Confirmation  Notice.  Ref. 
Ares(2015)1648792 - 17/04/2015. 

European  Commission  DG  ECFIN  (2015b).  Macro-Financial  Assistance  to  Georgia 
Disbursement of the First Tranche, Information Note to the European Parliament and the 
Economic and Financial Committee, ECFIN.D2/JCZ/lg Ares(2015). 

Compliance Statement of the Government of Georgia, 03/04/2017. 

European  Commission  DG  ECFIN  (2017a).  Macro-Financial  Assistance  to  Georgia 
Disbursement of the Second Tranche, Information Note to the European Parliament and 
the Council. Ref. Ares(2017)2209157 - 28/04/2017. 

46 

 
 
European Commission DG ECFIN (2017b). European Union macro-financial assistance 
(MFA):  Disbursement  of  the  2nd  Instalment  of  EUR  13,000,000,  Confirmation  Notice. 
Ref. Ares(2017)2617571 - 23/05/2017. 

BDO (Project funded by the European Union), Operational Assessment of the financial 
circuits and procedures in Georgia, November 2017. Specific Contract No.ECFIN‐ 139‐
2017/SI2.761024. 

47 

 
Annex 3: Timeline of the three Georgia MFA operations 

Date 

MFA progress 

22 October 
2008 

At the donors' conference held on 22 October 2008, 
the EU pledged to provide MFA to Georgia. 

MFA 
milestones 

Amount 

EUR 46 
million 

2009-2010 
13 January 
2011 

January 
2011 - 
August 2013 
12 August 
2013 

30 July 2014 

11 December 
2014 

December 
2014 

January 
2015 

April 2015 

November 
2015 

April 2017 

April 2017 

The first MFA (EUR 46 million, all in grants) was 
implemented. 
The EC adopted a proposal to provide up to EUR 
46 million of MFA-II to Georgia in the form of a 
grant of EUR 23 million and a medium-term loan 
of EUR 23 million, which was accompanied by a 
detailed evaluation of Georgia's needs (links to 
proposal and ex-ante evaluation). 
Delays on account of ‘procedural disagreement 
between the co-legislators’. 

Agreement was reached and the decision on MFA-
II was adopted by the European Parliament and the 
Council. (Decision No 778/2013/EU of the 
European Parliament and of the Council of 12 
August 2013 providing macro-financial assistance 
to Georgia). 

Stand-By Arrangement by the IMF approved. 
The Memorandum of Understanding (MoU) was 
signed outlining specific reform criteria attached to 
the assistance. 
Signature of the LFA and the Grant Agreement, 
ratification by the Parliament of Georgia of the 
MoU, LFA and the Grant Agreement. 
First tranche of the first EUR 13 million in grants 
in MFA-II programme to Georgia were disbursed. 

Second tranche of EUR 10 million in loans were 
disbursed. 

Review mission of MFA implementation 
succeeded; disbursement of the second tranche 
delayed due to the lack of progress reform 
programme implementation with the IMF. 
EC approved the release of the second loan tranche 
of EUR 23 million in light of the satisfactory 
progress of the Georgian authorities with 
implementing the policy conditionality under the 
MFA programme, as laid down in the MoU, and 
the IMF programme (following the compliance 
review on November 2015 and approving of a new 
USD 285 million arrangement under the EEF by 
the IMF Executive Board on 12 April 2017). 
Disbursement the grant element of EUR 10 million 
and loan element of EUR 13 million of the second 
instalment, thereby completing the MFA-II 
operation. 

48 

Proposal 

Decision 

MoU 

1st instalment 
grant 
component 
1st instalment 
loan 
component 

EUR 13 
million 

EUR 10 
million 

Progress 
review per 
MoU, 2nd 
instalment 
approved 

2nd instalment 
grant and 
loan 
component 

EUR 23 
million 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Date 

MFA progress 

September 
2017 

April 2018 

EC proposes new MFA 3 programme worth EUR 
45 million (of which EUR 10 million in grants). 
The EU Parliament and the Council adopted the 
Decision (EU 2018/598) providing further macro-
financial assistance to Georgia (MFA 3). 

Amount 

EUR 45 
million 

MFA 
milestones 

Proposal 

Decision 

August 2018  The MFA 3 Memorandum of Understanding 

MoU 

(MoU) was signed. 
1st instalment was disbursed. 

December 
2018 

1st instalment  EUR 20 
million 

49