COMMISSION STAFF WORKING DOCUMENT Evaluation Report Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL Evaluation of Decision N° 601/2015/EU of the European Parliament and of the Council of 15 April 2015 providing macro financial assistance to Ukraine
Date: 2021-04-09
Institutions: Directorate-General for Economic and Financial Affairs, European Commission
Subjects: European neighbourhood policy, International Monetary Fund, Russo-Ukrainian issue, Ukraine, economic reform, economic stabilisation, macro-financial assistance, public debt, structural adjustment

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 9.4.2021  SWD(2021) 73 final  COMMISSION STAFF WORKING DOCUMENT Evaluation Report Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL Evaluation of Decision N° 601/2015/EU of the European Parliament and of the Council of 15 April 2015 providing macro financial assistance to Ukraine {COM(2021) 169 final}  
 
 
Table of contents 

GLOSSARY ....................................................................................................................................................2 

1. 

INTRODUCTION .................................................................................................................................3 

Purpose of the evaluation 
Scope of the evaluation 

3 
3 

2. 

BACKGROUND TO THE INTERVENTION ......................................................................................4 

Economic situation 
Description of the intervention and its objectives 
Points of comparison 

4 
5 
7 

3. 

IMPLEMENTATION / STATE OF PLAY ...........................................................................................8 

 Implementation of the MFA operation 
 Economic situation 
MFA-III and MFA-IV 

8 
9 
10 

4.  METHODOLOGY............................................................................................................................... 11 

Evaluation techniques used 
Risks and limitations 

11 
12 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS ........................................... 13 

Evaluation Question 1: Relevance of the operation 
Evaluation Question 2: Effectiveness 
Evaluation Question 3: Efficiency of the operation 
Evaluation Question 4: EU added-value of the operation 
Evaluation Question 5: Coherence of the operation 
Evaluation Question 6: Social impact of the operation 
Evaluation Question 7: Public Debt Sustainability of the operation 

13 
15 
17 
18
19 
20 
21 

6. 

CONCLUSIONS .................................................................................................................................. 23 

ANNEX 1: PROCEDURAL INFORMATION ............................................................................................. 27 

Organisation, design and timing 

27 

ANNEX 2: STAKEHOLDER CONSULTATION STRATEGY .................................................................. 28 

       Mapping of stakeholder groups  

       Methods and tools for engaging with stakeholders 

           28 

           29 

ANNEX 3: METHODS AND DATA SOURCES ........................................................................................ 32 

ANNEX 4: TIMELINE OF THE UKRAINE MFA-III OPERATION……………………….…………….38 

1 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
             
 
 
 
Glossary 

Term  

             Meaning or definition 

AA 

AC 

 Association Agreement 

 Anti-corruption 

DCFTA 

 Deep and Comprehensive Free Trade Area 

DG ECFIN 

 Directorate-General for Economic and Financial Affairs 

DG NEAR 

 Directorate-General for Neighborhood and Enlargement Negotiations 

DSA 

EEAS 

EFF 

ENP 

EU 

FDI 

 Debt sustainability analysis 

 European External Action Service 

 Extended Fund Facility 

 European Neighbourhood Policy 

 European Union 

 Foreign Direct Investment 

FSDPL 

 Financial Sector Development Policy Loan 

GDP 

GNI 

IDPs 

IFIs 

IMF 

ISG 

MFA 

MoU 

NABU 

NAPC 

NBU 

PP 

SAPO 

SBA 

SOEs 

SWD 

UAH 

USD 

WB 

WTO 

 Gross Domestic Product 

 Gross National Income 

 Internally Displaced Persons 

 International Financial Institutions 

 International Monetary Fund 

 Inter-Service Steering Group 

 Macro-Financial Assistance 

 Memorandum of Understanding 

 National Anti-Corruption Bureau 

 National Agency for the Prevention of Corruption 

 National Bank of Ukraine 

 Public Procurement  

 Specialised Anti-Corruption Prosecution Office 

 Stand-By Arrangement 

 State-owned enterprises 

 Staff Working Document 

 Ukrainian Hryvnia 

 United States Dollar 

 World Bank 

 World Trade Organization  

2 

 
 
 
1. 

INTRODUCTION 

Purpose of the evaluation 

This  Staff  Working  Document  (SWD)  presents  an  evaluation  of  the  third  Macro-Financial 
Assistance  (MFA-III)  operation  to  Ukraine,  provided  by  the  European  Union  (EU)  over  the 
period  2015-2017.  It  largely  draws  on  an  independent,  ex-post  evaluation1  conducted  by  an 
external contractor, and a consultation process that involved targeted stakeholders in Ukraine as 
well as EU staff.  

The  aim  of  the  evaluation  is  to  assess  whether  the  MFA-III  operation  met  its  objectives  to 
support  Ukraine  in  addressing  its  balance-of-payments  problems  and  implementing  economic 
and  structural  reforms  that  would  stabilise  its  economy  and  enhance  the  sustainability  of  its 
external  position.  The  purpose  of  the  evaluation  is  to  support  decision-making  by  identifying 
areas  of  improvement  for  similar  on-going  or  future  MFA  operations,  while  also  ensuring 
transparency and accountability. 

Scope of the evaluation 
MFA  is  a  policy-based  financial  instrument  of  untied  and  undesignated2  balance-of-payments 
support  to  partner  countries.  It  is  designed  to  assist  third  countries  that  are  geographically, 
economically and politically close to the EU. MFA takes the form of medium/long-term loans, 
grants or a combination of the two. Unlike other, regular forms of aid provided by the EU, MFA 
is exceptional in nature and is mobilised on an ad-hoc case-by-case basis. Its objective is to help 
restore  a  sustainable  external  financial  situation,  while  encouraging  economic  adjustment  and 
structural  reforms  in  the  partner  country.  MFA  always  complements  (and  is  conditioned  on) 
financing provided in the context of a reform programme agreed with the International Monetary 
Fund (IMF). 

This evaluation covers the third MFA programme for Ukraine, implemented in 2015-2017. Two 
preceding MFA operations for this country, disbursed in 2014 and early 2015 for a total of EUR 
1.61 billion, were covered by a separate evaluation3.  

In the context of a deep economic recession and a severe confidence crisis due to the protracted 
armed  conflict  in  the  Eastern  part  of  the  country,  Ukraine  requested  additional  financial 
assistance from the EU and other international partners in late 2014.  In April 2015, the European 
Parliament  and  the  Council  adopted  a  decision4  to  provide  EUR  1.8  billion  of  MFA-III  to 
Ukraine as part of a wider package of international assistance. The objectives were to (i) alleviate 
Ukraine’s  external  financing  constraints,  balance  of  payment  and  budgetary  needs,  which 

1     Ex-post evaluation of the third Macro-Financial Assistance operation to Ukraine over the period 2015-2017, 

available at: https://ec.europa.eu/info/evaluation-reports-economic-and-financial-affairs-policies-and-spending-
activities/ex-post-evaluation-mfa-operations-ukraine_en 

2      In terms of the use of funds. 
3      Ex-post evaluation and Commission SWD on MFA-I and II to Ukraine are available at:  

https://ec.europa.eu/info/evaluation-reports-economic-and-financial-affairs-policies-and-spending-activities/ex-
post-evaluation-mfa-operations-ukraine_en 

4      Decision (EU) 2015/601 of the European Parliament and of the Council of 15 April 2015 providing macro-

financial assistance to Ukraine: 

        https://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1476965532409&uri=CELEX:32015D0601 

3 

 
 
                                                           
remained sizable even after the effects of MFA-I and II5 and other donor programmes; and (ii) to 
support the ambitious reform programme initiated under the first two MFAs. The assistance was 
to  be  disbursed  in  three  instalments  of  EUR  600  million  each6,  alongside  IMF  assistance  and 
contributions  from  other  donors.  Beyond  some  political  and  economic  pre-conditions, 
disbursements  were  linked  to  the  fulfilment  of  thirty-six  specific  structural  reform  conditions 
related  to  six  policy  areas,  namely:  public  finance  management;  governance  and  transparency; 
business environment; energy sector; financial sector; and social safety nets.   
In accordance with Article 34(1) of the Financial Regulation7, MFA operations in third countries 
are  subject  to  an  ex-post  evaluation.  In  turn,  the  aforementioned  MFA  Decision  for  Ukraine 
stipulates that the European Commission is required to “submit to the European Parliament and 
to the Council an ex post evaluation report”. 

To this end, the Directorate-General for Economic and Financial Affairs (DG ECFIN)  engaged 
an external contractor to complete an independent assessment, which informs this Staff Working 
Document. The objectives of the ex-post evaluation were:  

1.  To  analyse  the  impact  of  MFA-III  on  the  economy  of  Ukraine  and,  in  particular,  on  the 

sustainability of its external position; 

2.  To assess the added value of the EU’s intervention. In general, the evaluation seeks to draw 

lessons with respect to the EU’s financial assistance, i.e. 
a.  Whether  the  ex-ante  considerations  determining  the  design  and  terms  of  the  operation 
were appropriate, taking due account of the economic, political and institutional context; 
and 

b.  Whether the outcome of the programme met the objectives. 

These areas were assessed along the following key evaluation criteria: relevance, effectiveness, 
efficiency, EU added value, and coherence with  both other EU policies and interventions from 
international donors.  In addition, the evaluation assessed the social impact of the MFA and the 
impact on the sustainability of Ukraine’s public debt. This is further specified in the Evaluation 
Roadmap8. 

2.  BACKGROUND TO THE INTERVENTION 

Economic situation 

Following the Russian annexation of the Crimean peninsula, and due to conflicts in the eastern 
part  of  the  country,  the  economic  and  political  environment  in  Ukraine  deteriorated  sharply  in 
2014, resulting in a fully-fledged currency, banking and balance of payments crisis. Uncertainty 

5      For an assessment of the main outcomes of MFA-I and II to Ukraine, please refer to the related ex-post 

evaluation and Commission SWD, available at:  https://ec.europa.eu/info/evaluation-reports-economic-and-
financial-affairs-policies-and-spending-activities/ex-post-evaluation-mfa-operations-ukraine_en 

6      The MFA-III operation expired on 13 January 2018 without the third and final disbursement, given the lack of 

sufficient progress in the implementation of some of the related conditions.  
7     https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32018R1046 
8     https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives/12118-Ex-post-evaluation-of-macro-

financial-assistance-to-Ukraine- 

4 

 
 
 
                                                           
and  instability  related  to  the  confidence  loss  associated  with  the  conflicts  translated  into 
restricted access to finance and drove the fall in investment activity and household consumption. 
As a result, real GDP contracted by 6.8% in 2014.  

Weak economic activity, coupled with higher interest outlays on foreign currency denominated 
debt  in  light  of  strong  currency  depreciation,  as  well  as  sizable  losses  of  tax  collection  in  the 
eastern  parts  of  the  country,  led  to  a  widening  of  the  budget  deficit  in  2014.  The  deficit 
worsened  despite  a  number  of  austerity  measures  introduced  by  Ukrainian  authorities  in  both 
March and July 2014. In the same year, a major additional drag on public finances came from the 
state-owned  oil  and  gas  company  Naftogaz,  whose  activity  was  negatively  affected  by  the 
depreciation of the hryvnia and the gas arrears to Russia. As a result, the State had to inject UAH 
103 billion into Naftogaz by November 2014, an amount representing 6.8% of the yearly GDP. 
In this way, the public finance deficit run by Ukraine in 2014, including the deficit of Naftogaz, 
exceeded 10% of GDP, leading to a sharp deterioration of the country’s public debt metrics.  

On  the  external  side,  the  depreciation  of  the  hryvnia,  coupled  with  weak  domestic  demand, 
contributed  to  a  significant  adjustment  of  the  current  account  at  first.  However,  exports 
performance, despite benefitting from the weaker currency and the trade preference provided by 
the EU, was hurt by disruptions of the production chains and growing trade tensions with Russia. 
As a result, the current account deficit remained at 4% of GDP in 2014. This was accompanied 
by sizeable private sector financial outflows, due to dwindling confidence in an environment of 
high geopolitical uncertainty9. In the context of a deepening economic recession and confidence 
crisis, the substantial official financial assistance provided to Ukraine in 2014 proved insufficient 
to  offset  capital  flight  and  to  stop  the  continuous  drain  on  reserves.  By  December  2014, 
international reserves had dropped to USD 7.5 billion (corresponding to  1.8 months of imports 
only). 

Description of the intervention and its objectives 

In  this  difficult  macroeconomic  context,  in  April  2014,  Ukraine  entered  into  a  USD  17  billion 
(800% of quota), two-year Stand-by Arrangement (SBA) with the IMF. Subsequently, in March 
2015, a four-year Extended Fund Facility (EFF) of USD 17.5 billion was signed between the two 
parties, replacing the preceding SBA. To complement Ukraine’s arrangements with the IMF, in 
2014  and  early  2015,  the  EU  pledged  two  macro-financial  assistance  (MFA  I  and  II) 
programmes, of a combined amount of EUR 1.61 billion.  

Given  the  worsening  economic  situation  and  continuing  conflict  in  the  eastern  Ukraine,  the 
Ukrainian  authorities  requested  in  late  2014  a  third  MFA.  Upon  an  ex-ante  evaluation10,  the 
European  Commission  proposed11  in  January  2015  up  to  EUR  1.8  billion  in  MFA  loans.  The 
European Parliament and the Council adopted the related Decision12 on 15 April 2015. The EU’s 

9     The net outflow from the financial account amounted to USD 4.8 billion in January-October 2014. Withdrawals 
of  bank  deposits  by  non-residents  were  the  main  factor  behind  the  capital  outflow.  Both  FDI  and  portfolio 
investments also recorded outflows in the period, although of a smaller size.  

10      Commission SWD/2015/0001 final, available at:  
        https://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1476965500936&uri=CELEX:52015SC0001 
11     https://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1476965442393&uri=CELEX:52015PC0005 
12     https://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1476965532409&uri=CELEX:32015D0601 

5 

 
 
                                                           
contribution through the third MFA was an essential part of programme financing to cover the 
identified  financing  gap  under  the  IMF  programme;  without  it,  the  gap  would  not  have  been 
filled and the IMF could not have proceeded with its own programme. 

Following the consultation of the Member States' Committee, a Memorandum of Understanding 
(MoU)13 and Loan Facility Agreement related to this assistance were signed, and the ratification 
by  the  Ukrainian  Parliament  took  place  in  July  2015.  Annex  4  of  this  report  summarises  the 
timeline of the operation.   

The assistance was meant to be provided in three tranches of EUR 600 million each, disbursed 
between July 2015 and December 2017. Disbursements were conditional to good progress under 
the  IMF's  EFF,  political  prerequisites  (the  respect  of  human  rights,  effective  democratic 
mechanisms,  including  a  multi-party  parliamentary  system,  and  the  rule  of  law),  and  to  the 
fulfilment  of  the  set  of  policy  conditions  specified  in  the  Memorandum  of  Understanding. 
However,  the  MFA-III  operation  expired  on  13  January  2018  without  the  third  and  final 
disbursement having been released, given the lack of sufficient progress in the implementation of 
some of the required conditions. The Commission formalised the cancellation in a decision dated 
18 January 201814.  

The general intervention logic, which illustrates the objectives and expected results of the MFA-
III  operation,  applicable  to  the  2015-2017  Ukraine  programme,  is  summarised  in  the  graph 
below. 

13     https://ec.europa.eu/economy_finance/international/neighbourhood_policy/doc/mou_eu_ukraine_en.pdf 
14     Commission Decision EU/2018/405 final. Available at:  

https://ec.europa.eu/transparency/regdoc/rep/3/2018/EN/C-2018-405-F1-EN-MAIN-PART-1.PDF 

6 

 
 
 
                                                           
Figure 2.1 Intervention logic of MFA operation 

Source: ICF 

Points of comparison  

The points of comparison, against which the MFA operation of 2015-2017 is assessed, refer to 
the  situation  in  Ukraine  (1)  prior  to  the  intervention,  on  the  one  hand,  and  (2)  during  and 
immediately after the implementation of the MFA operation, on the other hand. As noted in the 
previous  subsection,  which  describes  the  economic  situation  prior  to  the  MFA  operation, 
particular  attention  is  paid  to  the  external  sector  and  the  fiscal  situation  of  the  Ukrainian 
economy. The following section will describe the implementation of the   MFA-III 
operation, 
looking at the key developments in the policy reform areas supported by the programme, as well 
as in the economic situation of the country.  

7 

 
 
 
 
3.  IMPLEMENTATION / STATE OF PLAY 

Implementation of the MFA operation  

To  achieve  the  objectives  detailed  in  section  1,  and  as  per  the  usual  MFA  procedure, 
disbursements under this operation were tied to the fulfilment of the political pre-conditions15, as 
well as good progress with the implementation of the IMF programme. The disbursement of the 
second and third tranches was also subject to the fulfilment of a set of thirty-six country-specific 
policy conditions, specified in the MoU and related to six structural reform areas16. 

MFA loan was provided on highly favorable terms. The first instalment was characterised by a 
coupon of 0.25% and a maturity of 5 years, while the second instalment had a coupon of 0.75% 
and a maturity of 14 years. For both tranches, a bullet capital repayment (i.e. a lump sum of the 
full outstanding amount) was envisaged in the last year of maturity. With respect to the interest 
rates, no grace period was applied. 

Ukraine  satisfactorily  fulfilled  all  the  policy  conditions  necessary  for  the  release  of  the  second 
instalment,  with  the  exception  of  conditions  12  (on  social  safety  nets)  and  13  (on  trade  and 
customs), for which waivers were granted. These two conditions required Ukraine to: (1) ensure 
the  effective  provision  of  social  benefits  and  services  to  internally  displaced  persons  (IDPs) 
through  adequate  legislation  and  funding;  and  (2)  to  refrain  from  introducing  new  trade-
restricting or trade-distorting measures, in line with Ukraine’s World Trade Organization (WTO) 
obligations.  With  respect  to  the  first  condition,  the  waiver  concerned  the  suspension  of  social 
payments  to  around  500,000  IDPs  living  in  the  non-government  controlled  areas  in  Eastern 
Ukraine.  Meanwhile,  the  waiver  for  condition  13  concerned  a  wood  export  ban17  adopted  by 
Ukraine  in  April  2015,  which  went  against  the  standstill  commitment  on  trade-restricting 
measures. 

However, the third and last tranche was not released, due to four conditions that were not met by 
the Ukrainian government before the expiration of the MFA programme. More specifically: 

  Measure  5,  on  anti-corruption,  whereby  Ukraine  committed  to  setting  up  an  electronic 
asset  disclosure  system  for  public  officials  and  a  related  verification  mechanism.  Ukraine 
did  not  fulfil  this  important  condition,  as  the  required  verification  mechanism  was  not 
established. 

  Measure  6,  still  on  anti-corruption,  whereby  Ukraine  committed  to  (i)  putting  in  place 
mechanisms  to  verify  (post-registration  and  following  clear  criteria)  the  accuracy  of  the 

15      https://ec.europa.eu/info/business-economy-euro/economic-and-fiscal-policy-coordination/international-

economic-relations/macro-financial-assistance-mfa-non-eu-partner-countries_en 

16      For the detailed list of policy conditions, please refer to the Memorandum of Understanding, available at:  
         https://ec.europa.eu/economy_finance/international/neighbourhood_policy/doc/mou_eu_ukraine_en.pdf 
17      The wood export ban refers to the temporary 10-year ban on export of unprocessed timber put in place by     

Ukraine through the Law of Ukraine No 325-VIII of 9 April 2015. The law imposed a ban on the export of all 
timber except the pine tree as of 1 November 2015, and as of 1 January 2017 of the pine tree as well. 

8 

 
 
 
                                                           
information provided by companies; and (ii) enforcing compliance with this obligation. The 
country did not fulfil this condition, since the required system to ensure the verification of 
information submitted by companies was not implemented.  

  Measure 17, on trade and customs, linked to the preceding condition 13. Ukraine did not 
fulfil this condition, as it did not repeal the export ban on unprocessed timber of November 
2015 and did therefore not refrain from introducing new trade-restricting measures in line 
with its obligations under the agreements of the WTO. 

  Measure 21, on the financial sector, whereby Ukraine committed to establishing a central 
credit registry. The country did not fulfil this obligation, since the necessary legislation to 
establish a central credit registry was not adopted.  

The  non-disbursement  decision  heavily  depended  on  the  two  unmet  conditions  on  anti-
corruption.  The  main  reasons  for  not  implementing  the  two  measures  within  the  agreed 
timeframe  were:  (1)  a  lack  of  political  will;  (2)  entrenched  bureaucracy,  which  slowed  the 
process;  and  (3)  vested  interest  of  some  stakeholders.  No  evidence  has  been  found  regarding 
negative  impacts  on  our  bilateral  relations  originating  from  the  non-disbursement.  Indeed,  the 
collaboration  between  the  Commission  and  the  Ukrainian  authorities  remained  positive  and 
constructive,  with  greater  focus  of  the  Ukrainian  authorities  on  the  implementation  of  the 
conditions and a more honest policy dialogue between the two sides.  

Therefore,  in  light  of  the  flexibility  already  granted  with  the  two  previous  waivers  and  the 
prominent  role  given  by  the  EU  to  transparency  and  the  rule  of  law,  the  EU’s  credibility 
appeared to be at stake and the non-disbursement decision is deemed justified.  

Some achievements in the policy areas tackled by MFA’s conditionality deserve particular note. 
In  the  area  of  public  finance  management,  MFA-III  policy  conditionality  supported  the 
implementation  of  an  online  public  procurement  platform  (ProZorro)  by  2016,  ensuring  open 
access  to  public  procurement  in  Ukraine  and  increasing  transparency  in  public  finance 
management.  Regarding  governance  and  transparency,  MFA  conditions  have  helped  in  the 
adoption of legal foundations for establishing independent and professional supervisory boards at 
state-owned  enterprises.  With  respect  to  the  energy  sector,  thanks  to  MFA-III  and  other 
international  donor  programmes,  the  sector  started  to  positively  contribute  to  the  State  budget, 
reversing the negative trend recorded between 2011 and 2014.  

Economic situation during the implementation of MFA-III 

After a cumulated GDP contraction of about 16% in 2014-2015, Ukraine returned to growth in 
2016 (+2.4% of GDP) as investment activity and household consumption started recovering. The 
recovery  continued  into  2017  and  2018,  with  GDP  growth  amounting  to  2.5%  and  3.3%, 
respectively. Inflation also started to decelerate. However, the depreciation of the local currency 
in  the  final  months  of  2017  contributed  to  soaring  prices,  resulting  in  inflation  of  13.7%  in 
December 2017, exceeding the 10% upper band of the central bank's inflation target. In order to 
tame inflation, the central bank reversed the monetary easing cycle in place since March 2015. It 
raised its policy interest rate by a cumulative 4.5 percentage points since October 2017, to 17% 
as of 2 March 2018. 

9 

 
 
With  respect  to  the  external  sector,  plummeting  domestic  demand,  partly  reflecting  the  sharp 
currency depreciation following the abolishment of the pegged regime by the National Bank of 
Ukraine, played a central role in reducing an unsustainable trade deficit and a rebalancing of the 
current  account  in  2015. However,  Ukraine’s current  account deficit started to  widen  gradually 
as of 2016 as a result of recovery in investment imports (reflecting growing business confidence) 
and  domestic  consumption.  Nonetheless,  a  further  widening  of  the  current  account  deficit  was 
contained  by  the  parallel  strong  rebound  in  exports  (17%  year-on-year  in  2017)  and  a  gradual 
increase in remittances from Ukrainians working abroad. The support from Ukraine's multilateral 
and bilateral partners, coupled with a current account adjustment and a gradual return of private 
financial flows, helped Ukraine replenish its international reserves to USD 18.8 billion at the end 
of 2017 despite weakness in FDI. 

Ukraine  has  also  made  significant  progress  in  the  consolidation  of  its  public  finances.  The 
overall fiscal deficit was reduced from 4.5% of GDP in 2014 to 2.3% in 2016 (compared with 
the 3.7% deficit target agreed for 2016 under the IMF programme). Budget execution continued 
to be strong in 2017, also due to robust tax collection and rising dividend payments from state-
owned enterprises. As a result, the consolidated state budget deficit in 2017 amounted to 1.4% of 
GDP. Thanks to the overall policy of fiscal consolidation, Ukraine also registered a reduction in 
gross government debt to an estimated 61.5% of GDP at the end of 2017 (down from the 69.2% 
of GDP at end-2016). 

Figure 3.1 Selected macro-economic indicators for Ukraine 

Indicator 

2014 

2015 

2016 

2017 

2018 

2019 

Real GDP change, % 

-6.6 

-9.8 

2.4 

2.5 

Consumer price inflation, %, end of 
period 
Key monetary policy rate, %, end of 
period 

Unemployment rate, % LFS 

General government balance, % of GDP 

Gross Public debt, % of GDP 

Current account balance, % of GDP 

International reserves, USD billion 

International reserves, month of imports 

17.5 

43.4 

12.4 

13.7 

14.0 

22.0 

14.0 

14.5 

18.0 

13.5 

9.3 

-4.5 

59.7 

-3.2 

7.5 

1.8 

9.1 

-1.6 

67.1 

2.3 

13.3 

3.0 

9.3 

-2.3 

69.2 

-1.3 

15.5 

3.0 

9.5 

-1.4 

61.5 

-2.2 

18.8 

3.2 

3.3 

9.8 

3.2 

4.1 

8.8 

-1.9 

52.2 

-3.3 

20.8 

3.3 

87.7 

1.8 

8.2 

-2.0 

44.3 

-0.8 

25.3 

4.0 

79.5 

1.6 

Gross external debt, % of GDP 

94.1 

129.5 

120.7 

102.9 

Foreign direct investment, % of GDP 
Sources: National Bank of Ukraine, Ministry of Finance of Ukraine 

0.2 

3.3 

3.5 

2.3 

MFA-III and MFA-IV 

The  non-disbursement  of  the  third  tranche  of  MFA-III  did  not  create  an  obstacle  for  the 
implementation of other EU programmes to Ukraine. Cooperation continued to be constructive 

10 

 
 
reflecting  also  that  a  new  MFA-IV  operation  of  up  to  EUR  1  billion  was  proposed  in  March 
2018, two months after the cancellation of the third disbursement under MFA-III. The operation 
was  meant  to  accompany  the  IMF’s  SBA  operation  of  USD  3.9  billion,  which  was  going  to 
follow the Extended Fund Facility ending in December 2018. The EU adopted the fourth MFA in 
July 2018. To underline the importance of anti-corruption measures and safeguard the credibility 
of  policy  conditionality,  the  first  disbursement  under  MFA-IV  was  only  made  in  December 
2018, after Ukraine fulfilled specific policy measures in relation to the unmet conditions of the 
previous  programme.  Notably,  as  part  of  the  implementation  effort  of  MFA-IV  first  tranche 
conditions, Ukraine made significant progress in the areas of anti-corruption policy18.  

4.  METODOLOGY 

Evaluation techniques used 

The methodology for evaluating the MFA operation in Ukraine over the period 2015-2017 was 
guided by the Commission’s Better Regulation Guidelines19 and the Guidelines for the Ex-Post 
evaluation of Macro-Financial Assistance Operations20. 

This  evaluation  was  supported  by  an  assessment  carried  out  by  an  external  contractor  from 
December 2019 until July 2020 and was overseen by an Inter-Service Steering Group (ISG). The 
external evaluation comprised of the following four phases: (1) inception, (2) data collection, (3) 
analysis and judgement, and (4) finalization and feedback.  

Evidence  and  the  data  were  collected  from  a  wide  range  of  sources  through  a  number  of 
complementary  quantitative  and  qualitative  approaches  aligned  to  the  consultation  strategy, 
including:  

(i) 

(ii) 

Desk research and macroeconomic data analysis of publically available documents 
from a range of sources (European Commission, IMF, World Bank, National Bank of 
Ukraine and Ukrainian Ministry of Finance);  
Forty  interviews  with  key  informants,  including  representatives  of  the  EU  and 
Member  State  political 
landscape,  Ukrainian  national  authorities  and  other 
stakeholders involved in the implementation or monitoring of MFA conditionality, as 
well  as  IFIs,  the  wider  donor  community,  civil  society  organisations  (CSOs)  and 
businesses  representatives.  Interviews  were  also  carried  out  with  the  European 
Commission (both in headquarters and the EU Delegation to Kyiv); 

(iii)  A  Delphi  survey  with  a  panel  of  experts.  Twenty-one  respondents  provided  valid 
feedback  with  a  high  degree  of  consensus  amongst  replies.  Further  details  on  the 

18  For  more 

information  about 

the  pre-conditions 

imposed  within  MFA-IV,  please 

refer 

to: 

https://ec.europa.eu/info/business-economy-euro/economic-and-fiscal-policy-coordination/international-
economic-relations/enlargement-and-neighbouring-countries/neighbouring-countries-eu/neighbourhood-
countries/ukraine_en 

19   European Commission, May 2015. Better Regulation Guidelines. Available at:  
       http://ec.europa.eu/smart-regulation/guidelines/toc_guide_en.htm. 
20   Available at: http://ec.europa.eu/dgs/economy_finance/procurement_grants/calls_for_tender/2015/015d/annex4-

methodological_orientations_en.pdf. 

11 

 
 
 
                                                           
Delphi survey (which focused on the macroeconomic impact of MFA-III, the impact 
of  structural  reforms  and  design  and  implementation  aspects  of  the  operation)  are 
included in Annex 221. 

(iv)  An  online  focus  group  with  attendees  from  Ukrainian  CSOs,  think  tanks  and 

(v) 

academia. Further details on the focus group are included in Annex 222; 
Two in-depth case studies on anti-corruption reform and social safety net reform as 
well as a third case study on the use of the euro in three MFA operations in Ukraine;  

(vi)  A stakeholder validation workshop to test and validate the emerging findings.   

Further  information  on  the  evaluation  techniques  is  available  in  Chapter  4  of  the  external 
evaluation  report.  Overall,  triangulation  of  findings,  obtained  using  different  techniques,  has 
helped  to  increase  validity  of  the  evaluation  results.  The  methodology  employed  is 
comprehensive and responded to the very specific and unique nature of the MFA operation23.  

Risks and limitations 

The  overall  reliability  and  validity  of  the  evaluation  is  strong.  Nonetheless,  a  number  of 
methodological limitations and challenges affected the evaluation:  

  While  generally  data  coverage  and  quality  is  good,  data  referring  to  some  key  national 

statistical indicators have been frequently updated recently, and some gaps exists.   

  Another  limitation  relates  to  the  reliability  of  the  judgment  provided  by  some 
stakeholders.  In a number of cases interviewees  (not directly involved in  the operation) 
had  limited  knowledge  of  specific  MFA-III  conditionality  or  were  unable  to  recall  the 
details of the operation and the general context. Additionally, the online format used for 
key  stakeholder  consultation,  Delphi  panel  and  focus  group  reduced  the  active 
participation and response rate of some actors involved.  

  As occurred for previous MFA evaluations, the changing economic environment over an 
extended period and the fact that the MFA-III operation was implemented in parallel with 
IMF  and  other  international  support  programmes  made  it  difficult  to  disentangle  the 
impact of the MFA from the impacts resulting from other factors. As a consequence, the 
counterfactual analysis was somewhat limited.  

Nonetheless, the identified risks and limitations do not put into question the overall reliability of 
the evaluation analysis, as they were mitigated by the fact that information was obtained from a 
wide range of sources, using different evaluation techniques, alternative scenarios and multiple 
rounds of feedback. Therefore based on the methods employed, the conclusions reached in  the 
evaluation are considered to be reliable and valid.  

21 Detailed results are available in Annex IV of the external evaluation report.  
22 Detailed results are available in Annex III of the external evaluation report. 
23    Further information concerning methods used to build the evidence for the evaluation is available in Annex 3 of 

the present Staff Working Document.  

12 

 
 
 
 
                                                           
5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS 

Relevance 

Question 1  

To  what  extent  was  the  MFA  operation  design  appropriate,  in  relation  to  the  outputs  to    
be produced and the objectives to be achieved? 

1.1 Was the size of the financial assistance adequate to address Ukraine’s financing needs? 

1.2 Were the form and timing of the financial assistance provided to Ukraine adequate?  

1.3 Was the conditionality of the MFA operation appropriate in relation to the objectives to be 
achieved? 

1.1  The  amount  of  the  EU  contribution  under  the  first  two  tranches  of  MFA-III  to  Ukraine 
corresponded to circa 0.7% and 0.6% of the country’s GDP in 2015 and 2017, respectively. In 
absolute terms, this was, by far, the largest single operation in the history of the MFA instrument, 
justified  by:  (1)  the  financing  needs  of  the  country  at  that  time;  (2)  the  size  of  the  country’s 
economy; (3) the important role of Ukraine for the stability of the European Neighbourhood; and 
(4) the importance attached to the integration of the country with the EU (in the context of the 
Association  Agreement  signed  in  2014,  following  the  “Revolution  of  Dignity”).  The  envelope 
covered 5% of Ukraine’s financing gap for the whole period 2015-201724, compared to the 7% 
initially  envisaged25.  Such  an  unprecedented  amount  of  MFA  has  further  increased  the 
importance of a careful design of the conditionality package and the implementation framework. 
However,  some  shortcomings  were  found  in  the  design  of  the  conditionality,  as  will  be 
highlighted below.  

Everything  considered,  the  EU  contribution  is  deemed  appropriate  (in  terms  of  burden  sharing 
with other donors) and proportional (necessary to achieve short-term macroeconomic stability in 
Ukraine).  

1.2  The  timing  of  MFA-III  was  particularly  relevant  to  ease  Ukraine’s  financing  needs  and 
balance of payments constraints. In particular, the disbursement of the first tranche in July 2015 
is deemed crucial for the country, given the high vulnerability of the Ukrainian economy at that 
time and the almost depleted foreign reserves. The EU decision-making process for MFA-III was 
extremely fast (the proposal submitted in January was adopted in April, with the MoU signed in 
May 2015). The second tranche disbursed in 2017 was not as critical, as the economy had begun 
to recover. 

Regarding the form of the operation, the MFA financing was provided in the form of EUR 1.8 
billion  loan,  on  highly  concessional  terms  that  could  not  have  been  directly  obtained  on  the 
market. The initial assessment of the main factors influencing the form of MFA instrument made 
the Commission excluding the possibility of a  grant  component26. Therefore, the adopted form 

24     Gap remaining after the contributions from the IMF and the World Bank. 
25     The difference between 5% and 7% was largely caused by the non-disbursement of the third tranche. 
26      European  Commission  (2011).  Criteria  for  determining  the  use  of  loans  and  grants  in  EU  Macro-Financial  

Assistance. Available at:  

13 

 
 
                                                           
(entirely loan) is considered and confirmed as appropriate. It was complementary to the EU grant 
support provided elsewhere by the EU to Ukraine in 2015-2017 totalling EUR 840 million.  

1.3 With respect to policy conditionality, MFA-III contained 36 conditions (46 if sub-conditions 
are  counted)  related  to  six  policy  areas:  (i)  public  finance  management;  (ii)  governance  and 
transparency;  (iii)  energy;  (iv)  financial  sector;  (v)  social  safety  net;  and  (vi)  business 
environment.  Given  the  relatively  short  amount  of  time  allowed  for  the  implementation  of  the 
reforms27, and given the level of ambition of most of the conditions concerned, it is believed that 
the number of reform conditions should have been lower. Although the decision to adopt such an 
unprecedented  number  of  conditions  may  reflect  the  exceptional  size  of  the  assistance  (more 
money  for  more  reforms),  it  also  distracted  the  authorities  from  focus  on  key  priorities  and 
required  substantial  human  resources  for  the  implementation  of  these  conditions.  The  non-
disbursement of  the third tranche was partially  explained by the high number of conditions,  as 
well as the design of the targets itself. Some reforms could have been broken down into smaller 
steps,  envisaging  a  gradual  approach  for  reform  implementation,  given  the  level  of  ownership 
and capacity of the Ukrainian authorities. This was particularly the case for condition 5 on asset 
declaration,  and  condition  6  on  beneficial  ownership.  The  high  level  of  ambition  of  the 
conditionality is nonetheless considered appropriate in most areas.  

The  thematic  areas  addressed  by  MFA-III  conditionality  were  relevant.  Many  of  them  were 
derived  from  the  EU-Ukraine  Association  Agreement  and  thus  aligned  with  the  country’s 
priorities.  However,  the  conditions  related  to  avoiding  an  introduction  of  new  trade-distorting 
measures proved problematic; trade-related disputes should and could be solved through trade-
specific channels and not as part of MFA.  

Overall,  the  design  of  MFA-III  programme,  both  in  terms  of  financing  envelope  and  focus  of 
reforms,  was  highly  relevant  in  view  of  the  targeted  objectives.  The  unprecedented  size  of  the 
disbursed assistance was crucial to support Ukraine in recovering from the economic crisis. The 
reform areas supported by the programme were highly relevant to the country's needs and well 
aligned  with  the  country’s  priorities.  However,  the  high  number  of  conditions  and  level  of 
ambition, coupled with the relatively short time horizon for implementation, reduced Ukraine’s 
ability  to  deliver  on  all  the  policy  reforms  and  limited  the  efficiency  of  the  overall  financial 
intervention.  More  sparing  use  of  conditionality,  with  reforms  milestones  broken  into  smaller 
targets, might have sustained a better motivation and feasibility throughout the operation. 

       https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52011SC0874&from=EN 
27     MFA-III initially meant to be completed by mid-2016. 

14 

 
 
 
 
 
 
 
                                                                                                                                                                                           
Effectiveness 

Question 2  

To what extent have the objectives of the MFA operation been achieved? 

2.1. To what extent has the MFA operation been effective in promoting macroeconomic stability, 
easing  external  financing  constraints  and  alleviating  Ukraine’s  balance  of  payments  and 
budgetary needs? 

2.2. To what extent has the MFA operation been effective in promoting structural reforms? 

2.1. The financial support provided under MFA-III has strongly contributed to the improvement 
of Ukraine’s economic performance from 2016. As described in section 3, Ukraine returned to 
modest  growth  in  the  first  half  of  2016,  started  to  consolidate  its  public  finance  and  regained 
access  to  the  international  debt  markets.  The  majority  of  the  experts  interviewed  in  the 
evaluation study believe that the macroeconomic outcomes would have been less favourable in 
the  absence  of  MFA-III.  In  particular,  the  highly  favourable  terms  of  the  loan  allowed  for  a 
reduction  in  the  cost  of  public  debt,  as  compared  to  alternative  sources  of  financing.  The 
magnitude  of  the  intervention  allowed  the  Ukrainian  government  to  pursue  fiscal  savings, 
without burdening public spending cuts and creating room for much-needed structural reforms. 
MFA conditionality helped in identifying key priorities for fiscal governance, guiding Ukraine in 
its recovery path.   

The results of the counterfactual analysis suggest that in the absence of the MFA’s first tranche 
in 2015, when Ukraine had no access to international debt markets, the most plausible course of 
actions  for  Ukrainian  government  would  have  been  fiscal  adjustment  in  the  form  of  drastic 
expenditure  cuts.  Domestic  borrowing  possibilities  were  severely  restricted  by  the  ongoing 
restructuring  of  the  banking  sector;  raising  revenue  through  higher  taxes  and/or  privatisation 
would have been similarly impossible.  Likewise, increased financing from  the  IMF  and World 
Bank would have been highly unlikely. However, substantial cuts in public spending could have 
jeopardised the country's path to economic recovery, with likely repercussions in terms of social 
tensions and political consensus. 

Had the second tranche in 2017 not been disbursed, the analysis shows that the most plausible 
course of action would have been to raise the required financing from the domestic debt market, 
which  had  already  recovered  by  then.  The  additional  cost  of  debt,  stemming  from  the  higher 
interest rates in domestic borrowing compared to MFA financing, would have reached USD 30 
million. 

Overall, under a scenario assuming no MFA-III and no IMF support to Ukraine, due to lack of 
access  to  international  markets  and  very  limited  domestic  funding  possibilities,  Ukraine  would 
have  most  likely  defaulted  on  its  debts  in  2015-2017,  with  hard-to-predict  but  certainly 
disruptive economic, social and political implications. 

Beyond financial contribution, MFA and other international donor programmes played a key role 
in renewing investors’ confidence and improving the country’s rating. This allowed Ukraine to 
regain  access  to  international  debt  markets  by  2017,  which  would  have  been  unlikely  in  the 
absence  of  the  above-mentioned  interventions.  Improved  business  confidence  contributed  to  a 

15 

 
 
strong  rebound  in  investment  activity  and  domestic  demand,  leading  to  economic  recovery. 
Furthermore,  the  support  granted  by  the  EU  and  other  international  partners,  coupled  with  a 
current  account  adjustment  and  a  gradual  return  of  private  financial  flows,  helped  Ukraine 
replenish its international reserves to USD 18.8 billion at the end of 2017.  

2.2 With respect to the short and medium-term structural effects of the MFA operation of 2015-
2017, there has been tangible progress beyond the fulfilment of the specific MFA-III conditions. 
Firstly, the energy sector is no longer causing budgetary problems to the Ukrainian government, 
helping the country to get on a path towards greater fiscal sustainability. Indeed, before 2014, the 
state-owned Naftogaz Group was a "black hole" in the Ukrainian government budget, with losses 
amounting  to  circa  6.2%  of  the  country's  GDP.  In  2019,  the  restructured  Naftogaz  group 
accounted  for  nearly  16%  of  total  revenues  of  the  State  budget28,  registering  a  remarkable 
reverse of trend since 2014.  

In the field of public finance management, the e-procurement system, ProZorro, has been widely 
acknowledged as a success, both domestically and internationally. Several steps have also been 
achieved in the field of taxation, including significant reduction of VAT refund arrears.  

Some  key  milestones  have  also  been  achieved  in  the  fight  against  corruption.  Prior  to  2015, 
Ukraine did not have dedicated institutions to fight corruption. The set-up of a new institutional 
architecture  has  therefore  been  a  major  undertaking  and  a  breakthrough  in  constructing  a 
systemic approach to tackle this phenomenon in the country. Given the early stage of Ukraine’s 
democratic transition, conditionalities agreed with the Ukrainian government required a split of 
competences among separate institutions, rather than opting for a single, central anti-corruption 
body.  As  part  of  MFA  conditionality,  the  National  Anti-Corruption  Bureau  (NABU),  the 
National  Agency  for  Prevention  of  Corruption  (NAPC)  and  the  Specialised  Anti-Corruption 
Prosecution  Office  (SAPO)  were  established.  Overall,  NABU  can  be  seen  as  a  successful 
example of a broadly independent body, despite some limitations. On the contrary, the other two 
institutions  still  lack  full  political  independence.  An  electronic  asset  declaration  system  for 
public  officials  also  entered  into  force  in  2016,  but  the  actual  verification  initially  lacked 
credibility,  scale  and  speed.  An  automatic  verification  mechanism  to  check  the  declarations 
started operating only in 2019, as part of MFA-IV.  

Overall, the EU MFA has proven to be essential for a consolidation of the country's balance-of-
payments  and  public  budget.  The  magnitude  of  the  intervention  allowed  for  fiscal  savings, 
necessary for the implementation of needed structural reforms. The simultaneous commitment of 
the EU and IMF has also allowed a renewed increase in investors and business confidence. With 
regard  to  structural  reforms,  MFA-III  was  effective  in  promoting  a  variety  of  measures  in  the 
areas  of  public  finance  management,  energy  policy,  business  environment,  financial  sector, 
social policy and anti-corruption, although some of these reforms required subsequent efforts at 
a later stage.  

28 

https://naftogaz.com/www/3/nakweben.nsf/0/A09B58DD11619020C22584F1002A61B6?OpenDocument&year
=2020&month=01&nt=News& 

16 

 
 
 
                                                           
Efficiency 

Question 3 

To what extent did the MFA operation design and implementation allow to carry out the 
intervention efficiently? 

3.1.  In  what  way  has  the  design  of  the  MFA  assistance  conditioned  the  performance  of  the 
operation in respect to its costs and its objectives? 

3.1 To answer to this question, some key issues need to be addressed: (i) the ownership of the 
programme  by  the  Ukrainian  authorities;  (ii)  their  capacity  to  undertake  reforms;  (iii)  the 
effectiveness  of  monitoring  activities;  (iv)  the  effectiveness  of  the  dialogue  among  concerned 
parties; and (v) the visibility of the MFA operation. 

The level of ownership of the reforms underpinning the MFA operation turned out to be more 
fragile  than  initially  anticipated  by  the  European  Commission  and  other  donors  between  2015 
and 2018. Ownership was uneven across the various Ukrainian authorities and bodies in charge 
of the implementation of the specific conditions: while it was consistently high for the Ministry 
of  Finance  and  the  National  Bank  of  Ukraine,  some  of  the  other  institutions  were  much  less 
involved. The factors explaining this drop in ownership were the resistance against some of the 
reforms  from  vested  interests,  also  in  the  Ukrainian  Parliament,  as  well  as  the  uneven 
performance  and  frequent  personnel  changes  in  many  parts  of  the  Ukrainian  public 
administration.  In  many  areas,  however,  reforms  were  promoted  thanks  to  the  diligent 
monitoring and the continuous pressure applied by the Ukrainian civil society on the successive 
governments. 

Even where present, capacity issues did not seem to cause the lack of progress on key reforms, 
as confirmed by the external  evaluation  report. However, the pressure placed on the Ukrainian 
authorities, due to the high number of conditions required in a limited time period, is believed to 
have caused an overloading of Ukrainian institutions. This may have led to consequent delays in 
some of the measures to be taken. On the other hand, the good collaboration among Ukrainian 
authorities and with the EU counterparts has been instrumental for a common understanding and 
commitment to the conditionality, as well as for a proper management of disbursements. 

Regular  progress  reports  submitted  by  Ukrainian  authorities,  missions  of  DG  ECFIN  staff  to 
Kyiv,  and  exchanges  with  experts  from  the  EU  Delegation  constituted  the  primary  tool  for 
monitoring  the  MFA  operation.  Given  the  complementarity  with  the  conditionality  of  other 
donor programmes, there was regular and close collaboration with other international actors (in 
particular with the IMF). This contributed to a proper conceptualization and implementation of 
the monitoring activity.  

Finally,  the  visibility  of  the  MFA-III  proved  to  be  limited,  as  was  the  case  for  other  MFA 
operations.  To  promote  MFA-III,  the  Commission  envisaged  press  conferences  and  press 
releases at the time of (i) the approval of the operation, (ii) the signature of the MoU, and (iii) the 
subsequent  disbursements.  The  press  releases  described  general  budget  allocations;  particular 
conditions included in the operation; progress made by Ukraine in achieving agreed milestones; 
and the release of MFA disbursements. Contributing factors to a low level of MFA-III awareness 
may include: (i) complexity of public finance and macroeconomic issues for the general public 

17 

 
 
and the way it was communicated; (ii) lack of publicity on MFA conditionality, financial aspects 
of the support and impacts of the programme; and (iii) confusion of MFA-III with other EU and 
IFI programmes. Feedbacks gathered from the key consulted stakeholders on this matter, during 
the  external  evaluation,  confirmed  the  limited  visibility.  These  findings  highlight  scope  for 
improvement for future EU interventions.  

Overall,  the  efficiency  of  the  MFA-III  operation  was  somewhat  limited  by  the  design  of  the 
conditionality  package  and  other  parallel  shortcomings,  related  to  reduced  ownership  and 
administrative capacity. The resistance from vested interest played a crucial role in the delays of 
many policy reforms, having relatively strong leverage over Ukrainian authorities. While there 
has been good collaboration between the EU and Ukrainian counterparts on the implementation 
process,  the  visibility  and  awareness  of  the  conditions  attached  to  MFA-III  proved  to  be 
relatively limited. 

EU added-value 

Question 4 

What  was  the  rationale  for  an  intervention  at  EU  level  and  to  what  extent  did  the  MFA 
operation  add  value  compared  to  interventions  by  other  international  donors?  Did  the 
operation  actually  lead  to  the  expected  impacts  and  added  value  of  international 
cooperation and what can be learnt for future operations? 

The  EU  intervention  primarily  aimed  to  provide  financing  to  Ukraine  at  a  time  of  crisis  when 
market-based  sources  of  funding  were  inaccessible.  MFA  was  an  indispensable  part  of  the 
overall  external  financial  assistance  package  put  in  place  by  the  international  community  in 
2015. With the highly concessional  terms  of the loans, MFA also  translated into fiscal  savings 
for the government and a gradual adjustment of the public deficit. The financial added value of 
MFA operations also derives from the fact that the EU could quickly mobilise and coordinate a 
wider  amount  of  resources,  as  compared  to  any  other  individual  donor  country29.  Beyond  the 
financial  contribution,  MFA  conditionality  granted  a  politically  reinforcing  effect  that 
contributed to the sustained mobilisation of the authorities around crucial (and often new) reform 
areas.  The  anti-corruption  field  and  public  procurement  reform  are  two  areas  where  the  EU 
stimulus was particularly helpful, as highlighted by the successful achievements reported before. 
Moreover, even for those areas in which the level of political ownership was  already high (i.e. 
public  finance  management  and  financial  sector),  international  support  played  a  key  role  in 
facilitating the policy process at Parliamentary level.    

Another (indirect) contribution of MFA-III  relates to  the increase of confidence among private 
investors.  The  MFA-III,  together  with  other  international  donors’  support,  helped  to  restore 
confidence in the economy, as confirmed by the majority of experts consulted in the external ex-
post  evaluation.  The  renewed  confidence  was  also  reflected  in  the  appreciation  of  the  hryvnia 
between  February  and  mid-March  2015  (time  when  the  IMF  programme  was  negotiated  and 

29      https://ec.europa.eu/neighbourhood-enlargement/neighbourhood/countries/ukraine_en 

18 

 
 
 
                                                           
concluded and the parameters of MFA-III were finalized), and in the improvement of Ukraine’s 
rating30.  

Overall,  the  ability  of  the  EU  to  mobilise  and  coordinate  significant  financial  resources  at  a 
relatively  quick  speed  was  a  key  rationale  for  this  intervention,  in  a  time  in  which  Ukraine’s 
financing  need  was  still  extremely  high.  MFA  conditionality  granted  a  politically  reinforcing 
effect  that  contributed  to  the  sustained  mobilisation  of  Ukrainian  authorities  around  crucial 
reform  areas.  The  programme  also  added  value  through  its  confidence-boosting  effect  on  the 
private sector.  

Coherence 

Question 5  

To what extent was the MFA operation in line with key principles, objectives and measures 
taken in other EU external actions towards Ukraine? 

5.1.  To  what  extent  was  MFA-III  coherent  with  the  broad  policy  framework  guiding  the  EU-
Ukraine relationship? 

5.2. To what extent the operation was coherent with the overall EU budget support package? 

5.3. To what extent it was coherent with other donors’ interventions (i.e. IMF, the World Bank)? 

5.1  The  implementation  period  of  MFA-III  (2015-2017)  coincided  with  the  entry  into  force  of 
the  Association  Agreement  (AA)31,  including  a  Deep  and  Comprehensive  Free  Trade  Area 
(DCFTA).  This  agreement  is  the  main  tool  for  bringing  Ukraine  and  the  EU  closer  together, 
promoting deeper political ties, stronger economic links and respect for common values. The AA 
was  signed  in  2014;  the  DCFTA  was  applied  since  January  2016  and  the  full  Association 
Agreement formally entered into force in September 2017. Title V, VI and VII of the AA outline 
a  framework  for  an  EU  intervention  in  support  to  Ukrainian  institutions,  financial  sector  and 
economic  activity.  Mapping  the  conditions  of  the  MFA-III  against  the  short-term  priorities 
established  in  the  AA  shows  that  MFA  conditions  were  well  aligned  with  the  key  priorities 
guiding EU-Ukraine relations.  

5.2  MFA-III  formed  part  of  the  wider  package  of  EU  support  to  Ukraine,  including  budget 
support, sectoral grant programmes and technical assistance that targeted largely the same areas 
as MFA-III (anti-corruption, public finance management, customs, public administration reform, 
energy). Over the period 2015-2017, EU deployed a total of EUR 840 million of grant support to 
Ukraine.   

Out  of  this  amount,  EUR  150  million  was  disbursed  as  budget  support  (most  of  it  as  State 
Building Contract - SBC), on top of the EUR 265 million from SBC disbursed in 2014. At the 
time  MFA-III  was  designed,  the  SBC  programme  was  in  place  but  the  benchmarks  for  the 
second tranche were yet to be met. MFA-III contributed with a reinforcing effect on several key 

30      Moody’s, 2014-2018. Ukraine. Available at: https://www.moodys.com/credit-ratings/Ukraine-Government-of- 

credit-rating-600037040?stop_mobi=yes 

31     https://trade.ec.europa.eu/doclib/docs/2016/november/tradoc_155103.pdf 

19 

 
 
 
                                                           
conditions, such as: the establishment of a verification system for asset declaration; the entry into 
force of the law on civil service; and the transparency around public procurement contracts.    

5.3 MFA-III conditions were complementary to the reform packages included in the programmes 
of  the  IMF  and  the  World  Bank  for  Ukraine.  For  instance,  in  the  field  of  public  procurement 
(PP), the IMF introduced a condition calling for the adoption of a new PP law; the World Bank 
also urged the adoption of a new PP law reducing the exemptions in competitive procurements. 
MFA-III then followed up with the introduction of an e-procurement system.  

In relation to reform of the National Bank of Ukraine (NBU), the EU put in place conditions on 
systemic banks and on related counterparties (MFA-III). The IMF tied its support to conditions 
related to internal control rules and governance and to autonomy framework of the NBU. This 
last condition was reinforced by the World Bank, which also introduced the implementation of a 
crisis management strategy (Development Policy Loan 1).  

On energy sector reform, the IMF pushed for the adoption of a new Gas Market Law and MFA-
III  for  the  secondary  legislation.  In  addition,  both  MFA  and  the  World  Bank  supported  a 
strengthening of the independence of the regulator. Some reforms were supported only by one of 
the  donors:  only  MFA-III  focused  on  social  safety  nets  for  IDPs,  while  monetary  policy  and 
pensions were only addressed by the IMF. 

Overall,  the  measures  of  MFA-III  were  completely  aligned  with  key  principles  and  measures 
taken in the EU external actions towards Ukraine. The conditions attached to the programmes 
supported reforms in many of the areas covered by the AA/DCFTA between the EU and Ukraine. 
In  addition,  the  operation  was  coherent  with  budget  support  commitments  taken  by  the  EU 
towards  Ukraine,  and  with  reform  packages  associated  with  the  IMF’s  and  World  Bank’s 
arrangements with the country.  

Social Impact 

Question 6  

What was the social impact of the MFA operation? 

6.1. Which were the direct effects of MFA conditionalities on the social situation in Ukraine? 

6.2.  Which  have  been  the  indirect  effects  of  MFA  operations  on  government  policies  and 
economic stability (as explored in the counterfactual analysis)? 

6.1  The  MFA  operation  included  four  conditions  having  a  direct  social  impact.  Two  of  them 
concerned a full functioning of the compensation schemes aimed at protecting poorer households 
from  increasing  energy  prices,  in  view  of  the  liberalisation  of  the  energy  market.  Two  further 
conditions  covered  an  effective  provision  and  evaluation  of  social  assistance  to  internally 
displaced  persons  (IDPs).  In  order  to  cushion  the  social  impact  of  the  energy  tariff  increases 
introduced since 2014, the housing utility subsidies were introduced in 2017 with their targeting 
subsequently  improved.  Moreover,  the  social  assistance  programme  for  IDPs  (the  temporary 
assistance  set  up  as  an  interim  measure  during  the  conflict  in  the  East)  was  made  permanent. 

20 

 
 
 
However, the suspension of social payments to many IDPs has damaged the living standards of 
many Ukrainian citizens, and resumption process was slow and inefficient.  

6.2  With  respect  to  the  indirect  social  impact  of  MFA  operation,  most  of  the  social  indicators 
such  as  average  wages,  households’  incomes  and  poverty  rate  have  improved  during  and  after 
the  MFA-III  operation;  the  unemployment  rate  slightly  increased  between  2015  and  2017,  but 
went down from 2018. For instance, nominal  wage growth and average household expenditure 
exceeded  consumer  price  inflation  in  that  period.  Obviously,  MFA-III  was  only  one  of  many 
factors influencing the social situation in Ukraine.   

As mentioned before, the results of the counterfactual analysis show that in  the absence of the 
first  tranche  of  the  MFA,  substantial  reductions  in  government  spending  would  have  been 
necessary. In particular, the government would probably have responded by delaying increases in 
public  pension  payments  (decreasing  the  value  of  these  payments  in  real  terms),  negatively 
affecting  household  resources  from  2015.  An  absence  of  the  second  tranche  conditionalities 
would likely have had a negative impact on the pace and depth of reform of social safety nets. 
This would have affected the social situation in Ukraine, considering the need for compensation 
mechanisms  to  protect  vulnerable  households  and  the  importance  of  the  resumption  of  social 
benefits and services for IDPs. Increased domestic financing, with limited social impacts, would 
likely have offset the absence of budgetary support (provided by MFA). 

Moreover,  an  absence  of  any  support  from  the  IMF  and  MFA  would  likely  have  created  a 
sovereign  debt  crisis.  Such  a  crisis  typically  blights  the  economy  with  issues  such  as  a  sharp 
economic  recession  with  corresponding  increases  of  the  unemployment  rate,  high  levels  of 
inflation  and/or  cuts  to  public  services  and  wages,  all  of  which  would  have  had  a  substantial 
negative effect on the social situation in Ukraine. 

Overall, MFA-III had a positive social impact on Ukrainian population, strengthening the social 
safety  net  and  protecting  low-income  households  against  higher  prices  resulting  from  energy 
reforms. The majority of social indicators also improved in this period.   

Public Debt Sustainability 

Question 7  

What was the impact of the MFA operation on public debt sustainability? 

7.1 To what extent has the MFA/IMF assistance contributed to returning the fiscal situation of 
Ukraine to a sustainable path over the medium to longer term? 

7.1  

Figure 5.7.1 macroeconomic variables and debt indicators for Ukraine in 2014-2017 

Indicator 

2014 

2015 

2016 

2017 

Real GDP change, % 

-6.6 

-9.8 

2.4 

2.5 

21 

 
 
 
Consumer price inflation, %, end of period 

Gross Public debt, % of GDP 

Public gross financing needs (% GDP) 

17.5 

70.3 

6.0 

43.4 

79.7 

12.5 

Combined fiscal deficit, incl. Naftogaz deficit (% 
GDP) 
Sources: National Bank of Ukraine, Ministry of Finance of Ukraine; IMF; Dragon Capital.  

10.0 

2.0 

12.4 

81.2 

12.9 

2.3 

13.7 

72.3 

9.4 

2.4 

According  to  data,  Ukraine’s  debt-to-GDP  ratio  increased  dramatically  from  40%  in  2013  to 
70%  in  2014,  80%  in  2015  and  81%  in  2016,  before  declining  in  the  following  years.  Such  a 
rapid increase in the ratio was largely driven by the depreciation of the hryvnia, as approximately 
62%  of  Ukraine’s  public  debt  was  denominated  in  foreign  currencies  in  2014.  As  the  hryvnia 
depreciated  against  the  US  dollar  by  33%  in  2014  and  by  a  further  46%  in  2015,  foreign-
currency denominated liabilities increased substantially in UAH terms. In addition, the combined 
public  finance  deficit,  including  the  deficit  of  Naftogaz,  exceeded  10%  of  GDP  in  2014.  The 
public  gross  financing  needs  relative  to  GDP  were  also  impacted  in  2015-2017.  The  ratio 
increased to 12.5% in 2015, primarily resulting from the currency depreciation, and it continued 
to rise in the following year, because of the increased financing needs related to resolutions and 
recapitalisations  in  the  banking  sector.  Both  indicators  (debt-to-GDP  and  financing  needs) 
improved in 2017 and the risk to debt sustainability moderated, thanks to the economic recovery, 
fiscal consolidation and the stabilization of the situation in the East.  

MFA-III  operation  played  a  key  role  in  smoothing  Ukraine’s  adjustment  path.  The  substantial 
financial contribution provided at highly favourable terms (in a period in which Ukraine was not 
granted  with  access  to  international  debt  markets)  allowed  for  fiscal  savings,  facilitating 
structural  reforms  and  social  spending.  In  the  absence  of  the  first  tranche  of  MFA-III,  the 
Ukrainian government would have been forced to make significant cuts in public spending. This 
would  have  led  to  an  initial  reduction  in  the  amount  of  public  debt,  but  could  be  politically 
difficult to accept and could jeopardize the upcoming reforms in the country. Had Ukraine not 
received the second tranche of MFA-III funding in April 2017, the most likely counterfactual  - 
borrowing the equivalent amount from domestic debt markets to fill the financing  gap  - would 
imply additional interest payments estimated at some USD 30 million over the next period, but 
no major risk to debt sustainability.  

In the scenario in which Ukraine did not receive funding from MFA-III nor the IMF, this would 
have severely impaired the country’s ability to attract  funding  from  domestic and international 
debt  markets and  would have decreased the availability of  assistance from  sources such  as the 
World Bank and the EBRD. As a result, Ukraine would most likely have defaulted on its debts 
over 2015-2017.  

Overall,  MFA-III  had  a  positive  direct  and  indirect  impact  on  Ukraine's  public  debt 
sustainability through a variety of  channels, including a signalling effect acting as catalyst for 
additional financial support and investor confidence. With the return of the economy to growth 
in 2016 and with quasi-fiscal deficits being nearly eliminated, Ukraine managed to put its debt 

22 

 
 
ratio on a clear downward path and reduce its public debt-to-GDP ratio to 72% at the end of 
2017. 

6.  CONCLUSIONS 

In April 2015, the European Parliament and Council adopted a third MFA operation of EUR 1.8 
billion  to  support  Ukraine.  The  EU  agreed  on  this  operation  relatively  swiftly  to  support  the 
country in addressing a growing balance-of-payment crisis, which was triggered by the economic 
and political crisis that followed the Russian annexation of the Crimean peninsula in 2014, and 
the subsequent conflicts in the eastern part of the country.  

This Staff Working Document is informed by an external ex-post evaluation conducted by ICF, 
in  collaboration  with  Cambridge  Econometrics,  which  covered  the  relevance,  efficiency, 
effectiveness, coherence, and EU value added of this MFA. It also explored the social impact of 
MFA  and  its  effect  on  Ukraine’s  public  debt  sustainability.  Limitations  encountered  relate 
predominantly to the changing economic environment over the period during which the MFA-III 
took  place,  and  the  fact  that  MFA-III  was  implemented  in  parallel  with  other  EU  and 
international  support  programmes  (making  it  hard  to  disentangle  the  impact  of  the  MFA 
operation specifically). Furthermore, the outbreak of the Covid-19 pandemic made it somewhat 
difficult  to  guarantee  an  active  participation  and  a  high  response  rate  of  key  consulted 
stakeholders  because  of  the  virtual  contacts  taken  with  online  focus  groups  and  interviews. 
Nevertheless,  these  limitations  do  not  put  into  question  the  overall  reliability  of  the  evaluation 
analysis  as  the  wide  range  of  sources,  the  use  of  different  evaluation  techniques,  alternative 
scenarios and multiple rounds of feedback mitigated them.   

Overall,  the  evaluation  found  that  MFA-III  was  highly  relevant  in  terms  of  its  objectives  and 
form (loans). The timing of disbursements and the unprecedented size of the envelope (EUR 1.8 
billion,  of  which  EUR  1.2  billion  disbursed)  have  largely  facilitated  Ukraine's  adjustment  path 
towards macroeconomic stability.  

MFA policy conditionality covered the most relevant reform challenges in Ukraine, namely: (i) 
public  finance  management,  (ii)  social  safety  net,  (iii)  financial  sector,  (iv)  governance  and 
transparency; (v) energy; and (vi) business environment.  

Out  of  three  MFA  tranches  foreseen  in  the  MoU,  two  were  disbursed  in  July  2015  and  April 
2017  respectively.  The  third  tranche  was  not  released  due  to  four  unmet  conditions.  Two 
conditions  were  on  anti-corruption,  where  Ukraine  did  not  manage  to  set  up  two  verification 
systems for asset disclosure and for the accuracy of the information provided by companies on 
beneficial ownership. A third condition was on trade, with Ukraine maintaining the export ban on 
unprocessed  timber.  The  last  condition  referred  to  the  financial  sector,  with  the  necessary 
legislation  to  establish  a  central  credit  registry  not  being  adopted  by  the  expiry  date  of  the 
operation. The main reasons for not implementing these measures within the agreed timeframe 
were: (1) a lack of political will; (2) the bureaucracy, which slowed the process; (3) a (too) high 
ambition level for some conditions, outside the main focus area of the reforms (for instance, the 
condition  on  beneficial  ownership)  and  (4)  vested  interest  of  some  stakeholders.  The  evidence 

23 

 
 
 
that  the  anti-corruption  conditions  were  met  at  a  later  stage  -  for  the  disbursement  of  the  first 
tranche of MFA-IV- highlights that the most complex reforms may require a longer time horizon 
for  implementation  than  the  two  and  a  half  years  foreseen  by  the  MFA.  Therefore,  particular 
attention is required while choosing the related targets, to ensure their feasibility already in the 
short term. In light of the flexibility already granted with two waivers for the disbursement of the 
second tranche and the prominent role given by the EU to fight corruption, the non-disbursement 
decision of the third tranche is deemed justified.  

to 

improve 

its  balance-of-payments  position.  In 

The  evaluation  found  that  MFA-III  was  effective  in  supporting  fiscal  consolidation  and  in 
helping  Ukraine 
the 
implementation  of  MFA-III,  Ukraine  returned  to  modest  economic  growth,  consolidated  its 
public finance and regained access to the international debt markets. The majority of the experts 
interviewed in the evaluation study believe that the macroeconomic outcomes would have been 
less favourable in the absence of MFA-III. In the counterfactual scenario without MFA-III, the 
Ukrainian authorities would most likely have had to further cut current budgetary expenditures, 
which could jeopardize the political acceptance of reforms. The simultaneous commitment of the 
EU and IMF has also allowed a return of investors’ and business confidence.    

the  period  of 

With  respect  to  structural  reforms,  MFA-III  was  in  general  effective  in  promoting  a  variety  of 
measures  in  the  areas  covered  by  the  programme.  Some  of  the  conditions  brought  particularly 
satisfactory  results,  such  as  setting  up  and  implementation  of  an  online  public  procurement 
platform (ProZorro) and turning the state-owned Naftogaz Group from a loss-making company 
into  the  main  contributor  to  the  State  budget.  On  the  other  hand,  the  implementation  of  some 
conditions met difficulties, especially in the field of anti-corruption and trade.  

The  efficiency  of  the  MFA-III  operation  was  somewhat  limited  by  the  complex  design  of  the 
conditionality package and a reduced ownership of conditionalities by the Ukrainian side. This 
limited ownership, leading to delays in the implementation of the agreed policy actions, can be 
attributed  to  (i)  limited  institutional  capacity;  (ii)  political  and  economic  instability  and  (iii) 
vested interests opposing the reforms.  A high number of conditions  (36 in  total), coupled with 
the relatively short time  horizon for implementation,  influenced Ukraine’s ability to  deliver on 
all the policy reforms and limited the efficiency of the overall financial intervention. As a result 
and  learning  from  this  operation,  the  European  Commission  envisaged  fewer  conditions  with 
reform  milestones  broken  into  smaller  and  achievable  targets,  for  the  following  MFA-IV 
operation  to  Ukraine.  While  there  has  been  good  collaboration  between  the  EU  and  Ukrainian 
counterparts  during  the  implementation  process,  the  visibility  and  awareness  of  the  conditions 
attached to MFA-III was limited. 

MFA-III  was  deemed  coherent  with  the  broad  policy  framework  guiding  the  EU-Ukraine 
relations, notably the Association Agreement. There is a high degree of consistency in a number 
of reform areas between EU budget support programmes and MFA conditionality. Furthermore, 
as far as external coherence is concerned, the MFA has not only contributed to ‘burden sharing’ 
with  the  International  Monetary  Fund  and  other  donors  in  financial  terms,  but  also  reinforced 
reforms  promoted  by  the  IMF  and  World  Bank  through  the  use  of  cross  conditionality  and 
complementary conditions. 

24 

 
 
The  ability  of  the  EU  to  mobilise  and  coordinate  significant  financial  resources  at  a  relatively 
quick speed was a key rationale for this intervention, at a time in which Ukraine’s financing need 
was  still  extremely  high.  MFA  conditionality  granted  a  politically  reinforcing  effect  that 
contributed  to  the  sustained  mobilisation  of  Ukrainian  authorities  around  crucial  reform  areas. 
The programme also added value through its confidence-boosting effect on the private sector.  

MFA-III  had  both  a  direct  and  indirect  impact  on  the  social  situation  in  Ukraine.  The  MFA 
operation  included  four  conditions  having  a  direct  social  impact,  which  concerned  functioning 
and better targeting of compensation schemes aiming at  protecting households from  increasing 
energy  prices,  as  well  as  provision  of  social  assistance  to  internally  displaced  persons  (IDPs). 
These conditions provided a tangible improvement in the situation of poorer households, in spite 
of  a  temporary  suspension  of  social  payments  to  many  IDPs  living  in  the  non-government-
controlled areas in Eastern Ukraine and their slow resumption. A counterfactual analysis shows 
that  in  the  absence  of  MFA  and  IMF  support,  the  indirect  effects  on  the  social  situation  in 
Ukraine would have been severe.  

Finally, MFA-III had a crucial role in improving the sustainability of Ukraine’s public debt. 
The  programme  allowed  for  fiscal  savings,  stemming  from  the  highly  favourable  terms  of  the 
loans, helping Ukraine to smoothen its adjustment path and to create fiscal space for reforms and 
sustained  social  spending.  MFA  had  also  a  signalling  effect,  acting  as  catalyst  for  additional 
financial support and investor confidence. 

In  conclusion,  MFA-III  was  highly  relevant  to  support  Ukraine  in  addressing  its  balance-of-
payment  problems  and  implementing  key  structural  reforms  to  stabilise  the  economy  and 
enhance  the  sustainability  of  its  external  position.  It  allowed  for  fiscal  savings  and  financial 
benefits, and acted as catalyst for additional financial support and investor confidence. The MFA 
conditionality  package  was  highly  relevant  to  the  country's  needs  and  fully  aligned  with  the 
related  IMF  programme.  It  created  a  politically  reinforcing  effect  that  contributed  to  the 
mobilisation of Ukrainian authorities around essential reforms, especially in areas not covered by 
other  international  donor  programmes.  However,  the  high  number  of  conditions  and  level  of 
ambition, coupled with the relatively short time horizon for implementation, hampered Ukraine’s 
ability to deliver on all the policy reforms and limited the efficiency of the overall financial intervention. 
More  sparing  use  of  conditionality,  with  reforms  milestones  broken  into  smaller  targets,  might  have 
sustained a better motivation and feasibility throughout the operation. 

Overall, the ex-post evaluation showed that MFA conditionality tends to be more effective when: 

1.  A high level of political ownership of the reform programme is ensured; 
2.  A  strong  alignment  with  other  EU  and  international  donors  programmes  is  foreseen,  especially 

for those interventions which appear to be more problematic;  

3.  Continuous technical assistance and advice is granted during the implementation process.  

A  number  of  the  conclusions  related  to  the  assessment  of  the  evaluation  criteria  will  require 
further  consideration  and  may  benefit  from  cross-comparison  with  other  completed  MFA 
operations. A meta-evaluation is planned for end of 2021 and will explore these points in more 
detail.  The  meta-evaluation  will  provide  a  synthesis  of  the  results  of  the  evaluation  studies  of 
MFA operations carried out in the last ten years and will consider the reliability and relevance of 

25 

 
 
 
the  methodology,  process,  outputs,  and  outcomes  of  all  the  MFA  operations  undertaken.  In 
addition, the exercise will assess the timeline of MFA operations and potential linkages to added 
value and operational visibility. 

26 

 
 
Annex 1: Procedural information 

A1.1 Organization, design and timing 

The  ex-post  evaluation  assessed  the  EUR  1.8  billion  MFA  operation  in  Ukraine,  which  was 
approved in 2015 and implemented between 2015-2017. The assessment was in line with article 
34(1)  of  the  Financial  Regulation32  and  the  relevant  MFA  Decision33,  which  required  the 
European Commission to submit an ex-post evaluation report to the European Parliament and the 
Council. The objective of the evaluation was to draw lessons with respect to the EU’s financial 
assistance,  in  particular  the  design  and  implementation  of  the  programme  and  the  way  it 
contributed  to  achieving  macroeconomic  stabilisation  and  fostering  structural  reforms.  Apart 
from identifying areas of improvement for similar on-going or future possible interventions, the 
evaluation  also  aimed  at  ensuring  better  transparency  and  accountability  of  the  Commission’s 
activities.  

The  evaluation  looked  at  various  aspects  of  this  particular  EU  intervention  (relevance, 
effectiveness,  efficiency,  EU  added  value,  coherence  with  other  EU  policies  towards  Ukraine, 
social impact and the impact on the sustainability of Ukraine’s public debt). In order to ensure 
validity,  the  analysis  and  conclusions  are  based  on  the  evidence  obtained  using  several 
evaluation  methods  (documentary  review,  macroeconomic  data  analysis,  targeted  stakeholder 
interviews, case studies, focus groups, qualitative counterfactual analysis, a Delphi survey and a 
social-impact analysis). 

The  lead  DG  to  carry  out  and  manage  this  evaluation  has  been  the  Directorate  General  for 
Economic  and  Financial  Affairs  (DG  ECFIN).  DG  ECFIN  chaired  the  ISG  that  was  set  up  to 
manage  the  evaluation.  Apart  from  DG  ECFIN,  the  ISG  comprised  of  representatives  of  other 
Commission  services  (the  Secretariat  General  and  the  Directorate-General  for  Neighbourhood 
and Enlargement Negotiations) and the EEAS. 

The Decide planning entry for the ex post evaluation of MFA-III for Ukraine is PLAN2020/6383 
and  the  indicative  roadmap  for  the  evaluation  was  published  in  March  202034  to  seek  wider 
feedback.  In  the  context  of  the  framework  contract  for  the  provision  of  evaluation  services 
related to MFA programmes, in November 2019 the Commission awarded the specific contract 
to undertake the external evaluation to ICF Consulting Services Ltd.   

A kick-off meeting, where the ISG and the external contractor discussed the deliverables and the 
evaluation methods, took place in January 2020. This was followed by meetings on the inception 
and  interim  reports  in,  respectively,  February  2020  and  May  2020.  The  draft  final  report  was 
submitted in June 2020 with  updates provided for a final  version approved in  August  2020.  In 
addition to meetings, ISG members were continuously informed and consulted (via email and by 
phone) during the evaluation. The work of the external contractor was complemented by internal 
analysis from Commission services. 

32    https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32018R1046 
33    https://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1476965532409&uri=CELEX:32015D0601 
34    https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives/12118-Ex-post-evaluation-of-macro- 

financial-assistance-to-Ukraine- 

27 

 
 
                                                           
Annex 2: Stakeholder consultation strategy35 

The  stakeholder  consultation  strategy  was  developed  with  the  overall  objective  to  collect 
information and opinions on various aspects of the MFA operation in Ukraine, structured around 
the main evaluation criteria. It was designed to collect as much valuable and relevant information 
as  possible  from  various  groups  and  people  involved.  In  addition  to  the  information  collected 
through key document reviews, the information obtained via consultation was used to support an 
ex-post assessment of the design, implementation and impact of the MFA-III operation by acting 
as source of evidence to address each of the evaluation questions.  

The  consultation  was  developed  in  line  with  the  Better  Regulation  Guidelines  on  stakeholder 
consultation36  and  it  focused  on  (i)  extracting  recollections  from  the  period  in  which  the 
operation was designed and implemented, and also (ii) on collecting views on the period after the 
MFA had ended, to assess its impact and sustainability.  

Consultation tools were tailored to each targeted stakeholder group, to collect information most 
appropriate  to  their  knowledge.  These  tools  were  also  associated  with  different  aspects  of  the 
MFA operation in Ukraine. As for primary data collection activities, these principally included: 
(1)  a  Delphi  survey;  (2)  in-depth  interviews  with  key  (EU  and  Ukrainian)  stakeholders;  (3)  an 
online focus group with civil society organisations and other non-governmental stakeholders. In 
addition,  an  evaluation  roadmap  was  published  from  6th  March  –  3rd  April  2020  to  seek  wider 
feedback. Feedback received was considered as part of the evaluation process. 

A2.1 Mapping of stakeholder groups 

Consultation  was targeted to  specialists  –  either  people who have been closely involved in  the 
development  and/or  the  implementation  of  the  MFA-III  operation,  or  people  with  expert 
knowledge  in  the  areas  related  to  the  objectives  of  MFA-III.  They  were  representatives  of 
Ukrainian authorities and civil society, International Financial Institutions and EU institutions. In 
total,  101  people  were  consulted  through  a  Delphi  survey,  in-depth  interviews  and  an  online 
focus group. Consultees can be grouped across the following four groups:  

Ukraine public institutions: 

-  Ministry of Finance 
-  Ministry of Social Policy 
-  National Anti-Corruption Bureau 
-  National Bank of Ukraine 

Ukraine civil society organisations: 

35        For  more  information  about  stakeholder  consultation,  please  refer  to  Annex  I,  II,  III  and  IV  of  the  Ex-post 
evaluation  available  at:  https://ec.europa.eu/info/evaluation-reports-economic-and-financial-affairs-policies-
and-spending-activities/ex-post-evaluation-mfa-operations-ukraine_en 

36    https://ec.europa.eu/info/law/law-making-process/planning-and-proposing-law/better-regulation-why-and-

how/better-regulation-guidelines-and-toolbox_en 

28 

 
 
 
                                                           
-  NGOs  
-  Think tanks 
-  Academics 
-  Media 

Wider Donor community: 

- 
IMF 
-  EBRD 
-  World Bank 
-  OECD 
-  GIZ 
-  Credit rating agencies 
-  Banks 

European Union: 

-  European Commission 
-  European External Action Service 
-  Member State Permanent Representatives to the EU  
-  European business representatives 

A2.2 Methods and tools for engaging with stakeholders 

A2.2.1 Delphi Survey 

The Delphi Survey seek to establish views on the role and contribution of the MFA in achieving 
macroeconomic stability, easing external financing constraints and alleviating Ukraine’s balance 
of  payments  and  budgetary  needs.  Responses  gathered  from  the  Delphi  panel  fed  into  the 
analysis of the relevance, effectiveness, and efficiency of the MFA operation as well as the debt 
sustainability and social impact analyses.  

Twenty-one  respondents  provided  complete  answers  to  the  survey.  Respondents  included 
representatives  from  think  tanks,  academia,  banks,  credit  rating  agencies,  investment  funds, 
media and consulting. 

The  Delphi  Survey  was  carried  out  on  the  basis  of  structured  questionnaires.  Specifically, 
participants were asked to elaborate on the plausible scenario(s) had (i) the EU MFA-III, or (ii) 
the joint MFA-IMF action not been implemented. As mentioned above, the Survey also covered 
aspects relating to the role of the MFA operation in promoting structural reforms and its social 
impact.  A  full  analysis  of  the  Delphi  survey  results  is  contained  in  Annex  4  of  the  external 
evaluation report37.  

37  Please, 

refer 

to:  https://ec.europa.eu/info/evaluation-reports-economic-and-financial-affairs-policies-and-

spending-activities/ex-post-evaluation-mfa-operations-ukraine_en 

29 

 
 
                                                           
A2.2.2 In-depth interviews 
ICF conducted forty semi-structured interviews with key stakeholders. An important aim of these 
targeted interviews was to explore selected stakeholders’ perspectives on the MFA operation in 
Ukraine.  Specifically,  detailed  information  was  gathered  on  the  operation,  including  (but  not 
restricted  to):  its  design  and  implementation;  perceived/realised  impact(s),  notably  its 
effectiveness  in  driving  reform  and  /  or  macroeconomic  stability;  its  ‘added-value’  when 
compared to other/similar support; and potential future improvements.  

The  objective  was  to  interview  various  officials  and  representatives  of  the  Ukrainian  national 
authorities (and other public entities involved in the implementation of the operation), the wider 
donor  community  and  the  European  Commission.  Additional  interviews  were  conducted  with 
representatives  of  the  business  community,  the  research  sphere  and  the  general  population 
(notably in the recipient country). The main focus of each interview varied by stakeholder type. 
Annex 2 of the external evaluation report38 provides details on the main points of discussion by 
stakeholder group mapped across the evaluation criteria.  

A2.2.3 Focus group 
An online focus  group discussion with  non-government/private sector representatives, based in 
Ukraine, was organised.  

The  aim  of  the  focus  group  was  to  capture  additional  insights,  perspectives,  and  thoughts  in 
relation  to  the  MFA  operation  in  Ukraine.  The  discussion  was  steered  to  specific  topics, 
including (but not restricted to): 

A.  Added  value  of  MFA-III  as  part  of  the  broader  package  of  assistance,  in  terms  of  (i) 

visibility of the assistance; (ii) attitudes towards the EU among the general public;  

B.  Issues  related  to  non-disbursement  of  the  third  tranche  (awareness,  justification  level, 

economic and political implications);  

C.  EU and MFA-III’s contribution to Ukraine’s overall reform efforts;   

D.  Three  in-depth  discussions  of  specific  reform  areas:  anti-corruption  and  judiciary; 

business environment; public administration reform.   

A  summary  of  the  focus  group  discussion  is  included  at  Annex  3  of  the  external  evaluation 
report39.  

38  Please, 

refer 

to:  https://ec.europa.eu/info/evaluation-reports-economic-and-financial-affairs-policies-and-

spending-activities/ex-post-evaluation-mfa-operations-ukraine_en 

39  Please, 

refer 

to:  https://ec.europa.eu/info/evaluation-reports-economic-and-financial-affairs-policies-and-

spending-activities/ex-post-evaluation-mfa-operations-ukraine_en 

30 

 
 
                                                           
A2.2.4 Stakeholder validation workshop 
Towards the  end of the  evaluation  process,  a stakeholder validation workshop was held  to  test 
and validate the emerging findings with stakeholders closely involved in the negotiation and/or 
implementation of MFA-III. The workshop consisted of a short presentation of the main findings 
and conclusions, followed by discussion and feedback.  

Participant groups included: 

 EU: DG ECFIN, SGUA, EEAS, EU Delegation in Kyiv   

 Ukraine: NBU, Ministry of Finance  
 IMF representative  

31 

 
 
 
Annex 3: Methods and data sources 

An evaluation matrix was developed to guide the choice and design of specific research methods, 
as  well  as  to  provide  a  framework  for  subsequent  data  analysis  and  interpretation.  The  table 
below provides  a high-level  overview of the data collection methods  and  analytical  techniques 
that were used to address each evaluation criteria. Further details are provided in the sub-sections 
that follow.  

Table A3.1 Overview of the methods and techniques used for the evaluation 

Relevance  Effectiveness  Efficiency  Coherence 

EU added 
value 

Documentary review 

Macroeconomic data 
analysis 

Key informant / 
stakeholder interviews 

Online Focus group 

Delphi survey 

Case studies (structural 
reforms) 

●●● 

●●● 

●●● 

●● 

●● 

●● 

Qualitative counterfactual 
analysis 

● 

Social impact analysis 

Debt sustainability 
analysis 

●● 

●●● 

●●● 

●● 

●● 

●● 

●● 

●●● 

●●● 

●●● 

●●● 

●●● 

●●● 

●●● 

●●● 

●●● 

●● 

● 

●● 

● 

●● 

● 

●●● a very important method for addressing the evaluation criterion 
●● an important method for addressing the evaluation criterion  
● a complementary method 

A3.1 Documentary review 

All relevant literature and official documentation published by European Institutions, Ukrainian 
authorities  and  other  international  organizations,  such  as  the  IMF  and  the  World  Bank  was 
reviewed to inform the evaluation.  

Table  A3.2  below  provides  an  overview  of  the  sources  and  types  of  documentary  evidence 
assembled  and  reviewed,  while  table  A3.3  details  key  macroeconomic  indicators  and  data 
sources used.    

32 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Table A3.2 Documentary sources of evidence for the evaluation 

Type 

Description 

European Commission files relating to 
the MFA-III operation 

EU policy-related documentation 

  Council Decision 2015/601/EU;  
  Ex ante assessment; 
  MoU for MFA-III; 
  Operational assessment; 
  Preparatory documents submitted to the Economic and Financial Committee; 
  Documentation on other EU assistance to Ukraine (State Building Contract 
(SBC) documentation and evaluation report) and the overarching policy 
framework (including annual report on implementation of the AA). 

  Association Agenda;  
  Progress reports on Association Agenda;  
  Reports on the implementation of the European Neighbourhood Policy (ENP) 

and the Visa Liberalisation Action Plan;  

  Reports on financing spent on EU sector budget support;  
  Technical assistance project descriptions. 

Documentation published by the 
Ukrainian authorities  

  Economic strategies;  
  Reform programme action plans;  
  Newswires. 

IMF documents  

World Bank data and documents  

Other 

  Letters of Intent submitted by Ukrainian authorities to the IMF;  
 
IMF country reports, reviews;  
  Some IMF research publications; 
  Relevant evaluations, including the ex post evaluation of the 2015 EFF 

programme. 

  Country Partnership Strategies;  
  Programme documents relating to the Bank’s Development Policy Loan 2 and 
Financial Development Policy Loan (DPL) and Financial Sector Development 
Policy Loan (FSDPL) 1 & 2;  

  Documents relating to projects supporting relevant reforms. 

  Data and reports prepared by other IFIs (e.g. EBRD) and key bilateral donors;  
  Reports on particular reforms in Ukraine from civil society organisations; 
  Local research publications provided by Ukrainian think tanks and non-

governmental organisations (NGOs), e.g. IER; 

  Academic and grey literature on political and economic developments in Ukraine 

and its progress with the implementation of structural reforms; 

  Reports produced by major credit rating agencies. 

33 

 
 
  
 
Table A3.3 Key Macroeconomic Indicators and Data Sources 

Component 

Data type 

Description 

Key data source(s) 

The real economy 

National accounts 

Indicators of macroeconomic  

performance 

Ministry of Finance, IMF 

Balance of 
payments 

Balance of 
payments statistics 

Indicators of external 
sustainability and trade 
conditions 

IMF 

Government 

Government 
finance statistics 

Financial System 

Monetary statistics 

Indicators of the government’s 
fiscal sustainability 
(expenditure, budget balance, 
debt, tax revenue etc.)  

Ministry of Finance, IMF, 
World Bank 

Banking sector, financing 
condition, interest rates, foreign 
exchange data, etc. 

Ministry of Finance, NBU, 
IMF 

Labour Market 

Other economic 
statistics 

Indicators of socioeconomic 
performance 

Ministry of Finance, IMF, 
World Bank 

A3.2 Semi-structured interviews with key stakeholders 
Forty key informants were consulted for the purposes of the external evaluation40. The interviews 
aimed at gathering information on the design, implementation and results of the MFA operation 
of 2015-2017. Interviewees received a copy of the semi-structured questionnaire in advance that 
was  then  used  to  guide  the  discussion.  Interviews  were  held  with  relevant  staff  from  the 
European  Commission,  the  EU  Delegation  in  Kyiv,  officials  from  the  relevant  ministries  and 
agencies in Ukraine, as well as representatives from International Financial Institutions. Annex II 
of the present document describes the strategy followed for stakeholder consultation. 

A3.3 Focus group 
An  online  focus  group  discussion41  was  organized,  with  representatives  from  Ukrainian  civil 
society, think thanks and the Academia. It complemented and crosschecked information gathered 
from desk research and targeted stakeholder interviews.   

40   See the list of completed interviews in Annex I and II of the external evaluation report. 
41     See the summary note from focus group discussion in Annex III of the external evaluation report. 

34 

 
 
 
 
 
 
                                                           
A3.4 Delphi survey 

A Delphi survey was undertaken with  a panel  of 54 experts, representing the business  society, 
think thanks, financial and macroeconomic analysts, and the Academia. The scope of the survey 
was  to  assess  on  the  contribution  of  the  MFA  in  achieving  macroeconomic  stability,  easing 
external  financing  constraints  and  alleviating  Ukraine’s  balance  of  payments  and  budgetary 
needs42.  Experts were interviewed using a structured questionnaire and were asked to elaborate 
on plausible scenario would MFA-III not have been available, and the potential implications.  

A3.5 Case studies 

Two in-depth  case studies on MFA-promoted reforms  were developed in the areas of  (1)  anti-
corruption (AC) reform and (2) social safety net reform.  
The case studies addressed the following aspects: 
  Rationale behind the selection of specific MFA conditions in the above areas (for AC: #6 2nd 
tranche,  on  establishment  of  AC  bodies,  #5  3rd  tranche,  on  asset  declaration;  for  Social 
safety  net:  #12  2nd  tranche,  on  provision  of  assistance  to  IDPs,  #16  3rd  tranche,  on  an 
evaluation of social service delivery to  IDPs), as well as the relevance and added value of 
MFA conditionality; 

  Significance  of  MFA  conditionality  in  the  context  of  the  overall  need  for  reform  in  a 

particular thematic area/sector;  

  How  the  MFA  conditions  were  implemented  and  whether  the  authorities  encountered  any 
obstacles  in  implementing  these  conditions  (i.e.  lack  of  capacity,  political  or  public 
resistance to change etc.); 

  Role  and  contribution  of  the  MFA  in  promoting  reform,  including  identification  of  key 

‘causal links’; 

  Short, mid and long-term benefits of the MFA conditions. 

In  addition,  the  case  studies  attempted  to  draw  on  some  lessons  learned  from  the  design  and 
implementation of similar previous reforms by two donors, the World Bank and the IMF.  
The case studies were primarily based on desk research and stakeholder interviews, and took into 
account inputs from local economic experts. 

Case study on the use of the euro 

This case study examined whether the three MFA operations in Ukraine had any impact – via the 
above channels – in promoting the role of the euro in asset and debt management by Ukrainian 
authorities.  The  analysis  was  broken  into  two  components  and  covered  three  dimensions: 
descriptive, exploratory and explanatory (see Table A3.4). 

42   Detailed results are available in Annex IV of the external evaluation report.  

35 

 
 
 
 
                                                           
Table A3.4 Overview of the approach to case study analysis 

Analytical dimension 

Analytical component 

Descriptive  

Exploratory  

Explanatory 

How the use of the euro has 
evolved in asset and debt 
management activities of the 
Ukrainian authorities 

X 

X 

Descriptive statistics 
on use of the euro 

Factors 
influencing and 
driving these 
changes 

Role and contribution of the MFA - 
whether and how the MFA 
operations contributed to any of the 
observed changes in the use of the 
euro by Ukrainian authorities 

X 

X 

How MFA 
played a role 

Whether MFA 
played a role 

A3.6 Qualitative counterfactual analysis 

A qualitative counterfactual analysis was used by the evaluation team rather than a quantitative 
approach, due to the difficulties of isolating the effects of MFA from other interventions (i.e. the 
IMF programme, parallel EU interventions and supports from other donors) and other exogenous 
and/or unobservable factors. It applied a theory-based approach to draw inferences regarding the 
role  and  contribution  of  the  MFA  in  promoting  macroeconomic  stabilisation.  Information 
gathered through the other methods was used to deduce wat might have happened: in the absence 
of  the  first  MFA  disbursement  (alternative  1);  in  the  absence  of  the  second  disbursement 
(alternative  2);  in  case  the  third  tranche  would  have  been  released  (alternative  3);  and  in  the 
absence of both MFA-III and IMF’s EFF operations (alternative 4).  

A3.7 Social Impact Analysis (SIA) 

For the Social Impact Analysis, the evaluation analysed trends of key indicators prior to, during 
and after the MFA operation:  (i) wages,  (ii) social  services provided for  IDPs,  (iii) households 
expenditures, (iv) employment, (v) unemployment, (vi) pensions. Counterfactual reasoning was 
applied to deduce the extent to which the MFA operation contributed to the observed outcomes.  

A3.8 Debt Sustainability Analysis (DSA) 

The objective of DSA is to evaluate the contribution of the MFA-III operation (2015-17) towards 
the sustainability of Ukraine’s public debt. The DSA undertaken relies on modelling the key debt 
burden indicators and macroeconomic variables which affect the path of Ukraine’s debt and its 

36 

 
 
 
 
 
 
 
 
 
capacity to manage its debt sustainably. A debt sustainability analysis tool developed by the IMF 
and World Bank for countries that can access international financial markets was used to support 
the assessment43.  

The DSA tool was used to generate short-term and medium-term projections of the dynamics for 
debt sustainability indicators (the debt-to-GDP ratio and the gross financing need-to-GDP ratio) 
for years 2015-2017.  

43  IMF  Staff  Guidance  Note  for  Debt  Sustainability  Analysis  in  Market-Access  Countries,  available  at: 
https://www.imf.org/external/np/pp/eng/2013/050913.pdf.  For  the  methodology  of  DSA  calculations  in  the 
evaluation  of  the  MFA-III  operation  2015-2017  in  Ukraine,  please  also  refer  to  Annex  11  of  the  external 
evaluation report.  

37 

 
 
 
 
 
                                                           
Annex 4: Timeline of the Ukraine MFA-III operation 

Date 

MFA progress 

Table A4.1 

MFA 
milestones 

Amount 

September 
2014 

Ukrainian  authorities  official  request  for  Macro-Financial 
Assistance  (MFA)  addressed  to  the  European  Commission 
(EC). 

MFA-III request 
for assistance 

January    
2015 

The EC adopted a proposal to provide up to EUR 1.8 billion to 
Ukraine in the form of loans. 

MFA-III 
proposal 

EUR 1.8 
billion 

March       
2015 

Immediate disbursement of the IMF’s EFF 

EFF 

EUR 4.7 
billion 

April 2015 

The  Parliament  and  the  Council  adopted  the  Decision  (No. 
601/2015/EU) on 15 April 2015. 

MFA-III 
decision 

EUR 1.8 
billion 

May 2015 

Signature of Memorandum of Understanding (MoU) outlining 
the specific reform criteria attached to the assistance and Loan 
Facility Agreement (LFA). 

MFA-III  

MoU 

Entry into force of the Loan Facility Agreement (LFA); 

MFA-III  

July 2015 

Ratification by the Ukrainian Parliament of the MoU and LFA 

Ratification and 
entry into force 
of MoU 

38 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Date 

MFA progress 

July 2015 

Disbursement of the first instalment of EUR 600 million. 

July 2015 

Disbursement of the second tranche of IMF’s EFF 

September 
2016 

Disbursement of the third tranche of IMF’s EFF 

April 2017 

Disbursement of the fourth tranche of IMF’s EFF 

April 2017 

following 

Disbursement of  the MFA-III second instalment of EUR 600 
million, 
the 
satisfactory 
implementation  of  the  policy  conditionality  under  the  MFA 
programme,  as  laid  down  in  the  MoU,  and  the  IMF 
programme 

progress  with 

MFA 
milestones 

MFA-III  

1st loan 
instalment  

EFF 

1st loan 
disbursement 

EFF 

2nd loan 
disbursement 

EFF 

3rd loan 
disbursement 

Amount 

EUR 600 
million 

EUR 1.55 
billion  

EUR 888 
million 

EUR 938 
million 

MFA-III  

2nd  loan 
instalment  

EUR 600 
million 

September  
2017 

Ukraine’s return to the international bond market 

International 
Market access 

January 2018 

Decision of 3rd tranche of MFA-III cancellation by the EC 

MFA-III 
3rd tranche 
cancellation 

EUR 600 
million 

39