Special Report No 21 // 2015#Review of the risks related to a results-oriented approach for EU development and cooperation action #(pursuant to Article 287(4), second subparagraph, TFEU)
Date: 2015-11-17
Institutions: European Court of Auditors
Subjects: aid system, cooperation policy, developing countries, development aid, least-developed country, poverty, project evaluation

EN 

2015 

NO 21

Special Report  Review of the risks related 

to a results‑oriented 
approach for EU 
development and 
cooperation action

EUROPEAN
COURT
OF AUDITORS

 
EUROPEAN COURT OF AUDITORS 
12, rue Alcide De Gasperi 
1615 Luxembourg 
LUXEMBOURG

Tel. +352 4398-1

E-mail: eca-info@eca.europa.eu 
Internet: http://eca.europa.eu

Twitter: @EUAuditorsECA 
YouTube: EUAuditorsECA

More information on the European Union is available on the internet (http://europa.eu).

Luxembourg: Publications Office of the European Union, 2015

Print 
PDF 
EPUB 

ISBN 978-92-872-3435-3 
ISBN 978-92-872-3417-9 
ISBN 978-92-872-3372-1 

ISSN 1831-0834 
ISSN 1977-5679 
ISSN 1977-5679 

doi:10.2865/507985 
doi:10.2865/619824 
doi:10.2865/609385 

QJ-AB-15-019-EN-C
QJ-AB-15-019-EN-N
QJ-AB-15-019-EN-E

© European Union, 2015
Reproduction is authorised provided the source is acknowledged.

For any use or reproduction of the Annex on p. 41, permission must be sought directly from the copyright holder.

Printed in Luxembourg

EN 

2015 

NO 21

Special Report

Review of the risks related 
to a results‑oriented 
approach for EU 
development and 
cooperation action

(pursuant to Article 287(4), second subparagraph, TFEU)

Audit team

02

The ECA’s special reports set out the results of its performance and compliance audits of specific budgetary areas or 
management topics. The ECA selects and designs these audit tasks to be of maximum impact by considering the risks 
to performance or compliance, the level of income or spending involved, forthcoming developments and political and 
public interest.

This performance audit was produced by Audit Chamber III which specialises in external actions spending areas. The 
Chamber is headed by ECA Member Karel Pinxten who is also the Reporting Member of this report. The audit was led by 
Philippe Froidure, director of Chamber III, supported by Hubert Devillé, assistant to the director.

 From left to right: P. Froidure, K. Pinxten, H. Devillé.

Contents

03

Paragraph

I-V 

Executive summary

1-12 

1-5 

6 

1.  Introduction

1.1.  Why results matter

1.2.  What is a result?

7-12 

1.3.  The overall framework relative to EU development and cooperation results

13-19 

2.  Results policy and regulatory background

13-18 

2.1.  Policy background

14-15 

2.1.1.  High‑level development goals

16-17 

2.1.2.  High‑level forums on aid effectiveness

18 

19 

2.1.3.  EU communications

2.2.  Legal and regulatory background

20-22 

3.  Our review

23-79 

4.  Key risk areas

24-38 

4.1.  Formulation of objectives and indicators

24-26 

4.1.1.  Terminology

27-31 

4.1.2.  Document alignment and consistency

32-35 

4.1.3.  Setting SMART objectives and clear indicators

36-38 

4.1.4.  Defining a partner country’s commitment to implement reforms as an objective

39-46 

4.2.  Establishment of the results chain

39-42 

4.2.1.  Linking actions and results through a clear logical chain

43-46 

4.2.2.  Contribution versus attribution

47-50 

4.3.  Increased complexity due to the integration of cross‑cutting issues

Contents

04

51-57 

4.4.  Harmonisation between development partners’ instruments and frameworks

51-53 

4.4.1.  Alignment with partner countries’ results frameworks

54-55 

4.4.2.  Harmonising the instruments used by development stakeholders

56-57 

4.4.3.  Coping with the juxtaposition of different accountability structures

58-61 

4.5.  Weaknesses in evaluation and results reporting

62-65 

4.6.  Data consolidation and reporting

66-68 

4.7.  Data quality

69-73 

4.8.  Budgetary outturn

74-79 

4.9.  Changes in EU actions’ context

74-77 

4.9.1.  Risk management and choice of implementation modalities

78-79 

4.9.2.  Exogenous factors

80-83 

5.  Conclusions and recommendations

80-82 

5.1.  Conclusions

83 

5.2.  Recommendations

Annex  — 

 Comprehensive Evaluation Framework for Budget Support Evaluations

Reply of the Commission

 
 
Executive  
summary

05

I
Each year, the EU dedicates substantial resources 
to assisting the development of the world’s poor‑
est countries. In order to establish the credibility of 
development assistance, it is essential that the results 
achieved with these resources can be demonstrated.

IV
Our review showed that these risk areas had been cor‑
rectly identified by the Commission in a wide range of 
documents. It also showed the interest in the actions 
that are being undertaken by the Commission to 
launch an EU international cooperation and develop‑
ment results framework.

V
Nevertheless, further actions remain to be taken in 
order to harness the full potential of the Commission’s 
initiatives to improve EU development and coop‑
eration results. In this context, we have formulated 
a number of recommendations that the Commission 
should take into account.

II
Recently, there has been renewed interest on the part 
of the European Parliament and the Council in shifting 
the focus of EU development and cooperation policies 
from activities to results. Also, in December 2013, the 
Commission issued a Staff Working Document on ‘Pav‑
ing the way for an EU Development and Cooperation 
Results Framework’ and, in March 2015, presented the 
framework by issuing another Staff Working Docu‑
ment on ‘Launching the EU International Cooperation 
and Development Results framework’.

III
In our review, we identified nine key risk areas related 
to a results‑oriented approach for EU development 
and cooperation action.

1. Introduction

06

1 

SWD(2014) 235 final — ‘EU 
Accountability Report 2014 on 
Financing for Development 
— Review of progress by the 
EU and its Member States’.

2  http://ec.europa.eu/budget/

mff/figures/index_
en.cfm#documents, January 
2015.

3  Commission publication on 
'Multiannual Financial 
Framework 2014‑20 and EU 
Budget 2014 — The figures'.

4  Report on the evaluation of 

the Union’s finances based on 
the results achieved: a new 
tool for the European 
Commission’s improved 
discharge procedure 
— A7‑0068/2014,30 
January 2014.

1.1. Why results matter

01 
Substantial resources are devoted 
to assisting the development of the 
world’s poorest countries. While 
emphasising the fact that there is no 
single source of data on development 
finance, the Commission estimated in 
July 2014 that the cumulative resourc‑
es available to developing countries 
over the 10‑year period from 2002 
were in the region of 2 171 billion USD, 
including domestic and international 
public and private finance.

02 
As part of the commitment to attain 
the internationally agreed Official 
Development Assistance (ODA) target 
of an ODA/GNI ratio of 0.7 %, in 2013, 
collective EU (including Member State) 
net ODA spending was 56.5 billion eu ‑
ros, which is equivalent to an ODA/GNI 
ratio of 0.43 % of the EU’s collective 
GNI1.

03 
The amounts scheduled to be made 
available for development and co‑
operation over the 2014‑2020 period 
are 19 661 million euros through the 
EU general budget2, and 30 506 mil‑
lion euros for ACP countries through 
the European Development Fund3.

04 
In order to establish the credibility 
of development aid, in particular as 
regards the instruments used, aid 
delivery methods and the funds con‑
cerned, it is essential that the results 
achieved with this support can be 
demonstrated.

05 
Public accountability is also an im‑
portant issue. In a report published 
in January 2014, the European Parlia‑
ment pointed out that ‘activity‑based 
budgeting [was] still the fundamental 
principle when drafting the budget of 
the Union’, and regretted that some 
of the proposals for the 2014‑2020 
programming period remained ‘fun‑
damentally input‑based (expenditure 
oriented) and, therefore, still focused 
on compliance with the rules rather 
than on performance’. The Parliament 
also pointed out that the performance 
framework used by the Commission 
for the multannual financial framework 
2014‑2020 should encompass the fol‑
lowing three main elements4:

(a)  achievement of programme objec‑

tives (results);

(b)  sound programme management 
by the Commission and the Mem‑
ber States;

(c)  the contribution of programme 

results and sound management to 
the Union’s main objectives.

07

5 

For the purposes of this report, 
we use the terminology for 
these three categories of 
results as stated in external 
action financing instruments. 
Similar terms may be used 
differently in other EU 
documents.

1. Introduction

1.2. What is a result?

06 
A result is a measurable consequence 
deriving — directly or indirectly — 
from a cause and effect relationship. 
A results‑oriented approach to devel‑
opment and cooperation relies on the 
principle that the focus of develop‑
ment and cooperation interventions 
should be on the delivery of results, 
rather than on activity or process man‑
agement. Results usually cover three 
categories5:

(a)  output: what is produced or ac‑
complished with the resources 
allocated to an intervention (e.g. 
training courses delivered to un‑
employed people, roads built);

(b)  outcome: change that arises from 
an intervention and which nor‑
mally relates to its objectives. 
Outcomes may be expected or 
unexpected, positive or negative 
(e.g. trainees who have found em‑
ployment, improved accessibility 
to an area due to the construction 
of a road);

(c)  impact: longer‑term socioeco‑

nomic consequences that can be 
observed for a certain period after 
an intervention has been com‑
pleted, which may affect direct 
beneficiaries of the intervention or 
other indirect beneficiaries.

1 Notion of results
e
r
u
g
i
F

Inputs

Activities/
process

Outputs

Results

Outcome

Impact

‘set out the processes (…) taken 
forward by the Commission to deliver 
against the ‘Agenda for Change’ com-
mitment to strengthen its ability to 
monitor and report operational results 
achieved through the implementation 
of EU-funded development and coop-
eration projects and programmes’6.

6 

SWD(2013) 530 final of 
10 December 2013.

1.3. The overall 
framework of EU 
development and 
cooperation results

07 
Besides existing mechanisms aimed at 
measuring and reporting on results, 
a notable initiative taken by the Com‑
mission in December 2013 was the pro‑
duction of a Staff Working Document 
on ‘Paving the way for an EU Develop‑
ment and Cooperation Results Frame‑
work’. This initiative aimed to:

 
08

7 

8 

9 

SWD(2014) 200 final of 
26 June 2014 ‘Overview of the 
Monitoring, Reporting and 
Evaluation Frameworks for the 
MFF 2014‑2020 
Programmes — 
Accompanying the document 
Report from the Commission 
to the European Parliament 
and the Council on the 
evaluation of the Union’s 
finances based on the results 
achieved’.

European Parliament Decision 
of 3 April 2014 on discharge in 
respect of the implementation 
of the general budget of the 
European Union for the 
financial year 2012, Section 
III — Commission and 
executive agencies — ref. 
P7_TA‑PROV(2014)0287, 
section ‘External relations’, 
paragraph 264.

‘Council conclusions on an EU 
Development and 
Cooperation Results 
Framework’ — Council of the 
European Union Foreign 
Affairs (Development) 
meeting, 19.5.2014.

10  SWD(2015) 80 final of 
26 March 2015.

1. Introduction

08 
This Staff Working Document took into 
account the conclusions of a study, un‑
dertaken by the Commission in 2013, 
which aimed to provide an analysis of 
lessons learnt from five development 
agencies which were already imple‑
menting their own results framework. 
It presented a preliminary approach 
to drafting an overall EU development 
and cooperation results framework. It 
described such a framework as follows:

‘A results framework is a tool that 
is used by different development 
partners to measure results achieved 
against strategic development objec-
tives. The results framework can be 
seen as an articulation of the different 
levels, or chains, of results expected 
from the implementation of a particu-
lar strategy, programme or project’.

09 
The EU development and cooperation 
results framework will have two main 
purposes: as an accountability tool — 
in order to inform stakeholders of the 
results that have been achieved — and 
as a management tool — in order to 
provide performance data as a basis 
for management decisions.

10 
In June 2014, another Commission 
Staff Working Document was issued, 
aiming to describe ‘the framework for 
monitoring, evaluation and reporting 
to the end of the next multiannual 
financial framework that results from 
the legal instruments supporting the 
next generation of programmes and 
the additional work of the Commission 

to complete the framework’ — the 
‘MORE framework’. This Staff Working 
Document provides, for each policy 
area, an overview of the general and 
specific objectives of the instruments, 
the main indicators attached to each 
objective, monitoring arrangements, 
and the timing and content of evalua‑
tions and reporting7.

11 
In April 2014, the European Parliament 
expressed its support for ‘the Commis‑
sion’s continuing efforts to shift from 
an input‑based to a performance‑ and 
impact‑oriented approach’8. In May 
2014, the Council welcomed the Com‑
mission’s Staff Working Document of 
December 2013 on ‘Paving the way for 
an EU Development and Cooperation 
results framework’ and expressed its 
interest in the ‘results of the pilot‑
ing phase, with a view to the timely 
finalisation of the EU development and 
cooperation framework, so that report‑
ing on results achieved during 2014 
can be presented in the first semester 
of 2015’9.

12 
In March 2015, the Commission is‑
sued a new staff working document 
on ‘Launching the EU International 
Cooperation and Development Results 
Framework’. This document aimed 
to translate the approaches set out 
in the 2013 Staff Working Document 
(see paragraph 7) into operational DG 
International Cooperation and Devel‑
opment plans to report annually as 
of 2015 in order to demonstrate how 
funds spent contribute to the achieve‑
ment of the policy objectives10.

2. Results policy and 
regulatory background

09

11  http://ec.europa.eu/

europeaid/policies/international‑ 
development‑dialogue_en

12  Rio+20 outcome document 
‘The Future We Want’.

13  United Nations — General 

Assembly — Integrated and 
coordinated implementation 
of and follow‑up to the 
outcomes of the major United 
Nations conferences and 
summits in the economic, 
social and related fields — 
Follow‑up to the outcome of 
the Millennium Summit — ref. 
A/68/L.4.

14  United Nations — General 

Assembly — The road to 
dignity by 2030: ending 
poverty, transforming all lives, 
and protecting the planet — 
Synthesis report of the 
Secretary‑General on the 
post‑2015 sustainable 
development agenda — ref. 
A/69/700.

15  See also COM(2015) 44 final of 
5 February 2015 ’A Global 
Partnership for Poverty 
Eradication and Sustainable 
Development after 2015’.

Global discussions on the wider 
agenda beyond 2015 are currently 
ongoing through various processes. 
A Synthesis Report on the post‑2015 
sustainable development agenda was 
published by the UN Secretary‑General 
in December 201414,15.

2.1.2. High‑level forums on 
aid effectiveness

16 
The European Commission and EU 
Member States have taken part in 
several high‑level forums aiming to 
improve the effectiveness of develop‑
ment aid programmes:

(a)  the Rome Declaration on Harmoni‑

sation (2003);

(b)  the Paris Declaration on Aid Effec‑

tiveness (2005);

(c)  the Accra Agenda for Action 

(2008);

(d)  the Busan partnership for effective 
development cooperation (2011);

(e)  the Mexico First High‑Level Meet‑
ing of the Global Partnership for 
Effective Development Coopera‑
tion: Building Towards an Inclusive 
Post‑2015 Development Agenda 
(2014).

2.1. Policy background

13 
The role of the Commission on the in‑
ternational stage is set out in Article 17 
of the Treaty on the European Union. 
The Commission works, in particular, 
with the United Nations and the Or‑
ganisation for Economic Cooperation 
and Development (OECD) on devel‑
opment cooperation issues. In this 
context, high‑level development goals 
are agreed, together with principles 
for providing more effective aid11.

2.1.1. High‑level 
development goals

14 
In September 2000, the General As‑
sembly of the United Nations adopted 
the United Nations Millennium Decla‑
ration, which committed its members 
to a new global partnership with the 
aim of reducing extreme poverty and 
set a series of time‑bound targets — 
with a deadline of 2015 — which are 
commonly referred to as the Millen‑
nium Development Goals (MDGs).

15 
In June 2012, the Rio+20 Conference 
agreed to launch a process to develop 
a set of Sustainable Development 
Goals (SDGs), which will build upon 
the Millennium Development Goals12. 
In September 2013, world leaders 
renewed their commitment to meet 
the MDG targets and agreed to hold 
a high‑level summit in September 
2015 to adopt a new set of goals to 
build on the achievements of the 
Millennium Development Goals13. 

2. Results policy and regulatory background

10

17 
In these forums, a consensus emerged 
among donors and partner countries, 
and a number of principles relating 
to development results were agreed, 
including:

(a)  strengthening partner country 
governments’ ability to assume 
a greater leadership role and take 
ownership of development results;

(b)  working with partner countries to 
rely, as far as possible, on part‑
ner countries’ frameworks for 
results‑oriented reporting and 
monitoring;

(c)  harmonising donors’ monitoring 
and reporting requirements;

(d)  improving management for results 
by donors and partner countries;

(e)  increasing the focus on develop‑

ment results.

2.1.3. EU communications

18 
The European Union’s commitment to 
internationally agreed development 
goals and making aid more effective 
has been documented in several pub‑
lic statements and communications, 
all of them expressing, in one way or 
another, the importance of develop‑
ment and cooperation results:

(a)  Joint statement by the Council and 
representatives of the Member 
States’ governments on the Euro‑
pean consensus for development 
(ref. 14820/05 of 24 February 2006);

(b)  Supporting developing coun‑
tries in coping with the cri‑
sis (COM(2009) 160 final of 
8 April 2009);

(c)  A 12‑point EU action plan in sup‑
port of the Millennium Develop‑
ment Goals (COM(2010) 159 of 
21 April 2010);

(d)  Operational Framework on Aid Ef‑
fectiveness — General Secretariat 
of the Council (ref. 18239/10 of 
11  January  2011);

(e)  Increasing the impact of EU De‑

velopment Policy: an Agenda for 
Change (COM(2011)637 final of 
13 October 2011);

(f)  A decent life for all: Ending poverty 
and giving the world a sustain‑
able future (COM(2013) 92 final of 
27 February 2013);

(g)  Beyond 2015: towards a compre‑
hensive and integrated approach 
to financing poverty eradication 
and sustainable development 
(COM(2013) 531 of 16 July 2013).

2. Results policy and regulatory background

11

16  Accountability refers mainly to 
the democratic oversight of 
policies and activities of public 
bodies, whereas public audit 
refers to the financial and 
performance audits of policies 
and related public funds, and 
their link to the accountability 
process. See European Court 
of Auditors’ report on ‘Gaps, 
overlaps and challenges: 
a landscape review of EU 
accountability and public 
audit arrangements’, 2014 
(http://eca.europa.eu).

17  See also Article 163 of the 
Financial Regulation.

18  Regulation (EU, Euratom) 

No 966/2012 of the European 
Parliament and of the Council 
of 25 October 2012 on the 
financial rules applicable to 
the general budget of the 
Union and repealing Council 
Regulation (EC, Euratom) 
No 1605/2002 (OJ L298, 
26.10.2012, p. 1).

19  Similar provisions are 

contained in European 
Development Funds’ financial 
regulations.

2.2. Legal and regulatory 
background

19 
The main provisions concerning ac‑
countability for results and public 
audit16 can be found in the Treaty on 
the Functioning of the European Union 
(TFEU), in financial regulations for 
the EU general budget and European 
Development Funds, and in the regula‑
tion on common rules and procedures 
for the implementation of the Union’s 
instruments for financing external 
action.

(a)  The Treaty on the Functioning of 

the European Union provides that:

(i)  the Commission shall submit to 
the European Parliament and 
to the Council an evaluation 
report on the Union’s finances 
based on the results achieved 
(Article 318(2)); and

(ii)  the Court of Auditors may, at 

any time, submit observations, 
particularly in the form of spe‑
cial reports (Article 287(4))17.

(b)  The Financial Regulation applica‑
ble to the general budget of the 
Union (the Financial Regulation) 
stipulates the following18,19.

(i)  Specific, measurable, achiev‑

able, relevant and time‑bound 
objectives must be set for all 
sectors of activities covered 
by the budget. The achieve‑
ment of those objectives shall 
be monitored by performance 
indicators for each activity 
(Article 30(3)).

(ii)  In order to improve deci‑
sion‑making, institutions 
must carry out ex ante and ex 
post evaluations of all pro‑
grammes and activities which 
entail significant spending 
(Article 30(4)).

(iii)  The Commission must attach 
to the draft budget a number 
of documents, including ac‑
tivity statements containing in‑
formation on the achievement 
of all previously set specific, 
measurable, achievable, rel‑
evant and timed objectives for 
the various activities, as well 
as new objectives measured 
by indicators (Article 38(3)(e)
(i)), a full justification includ‑
ing a cost‑benefit analysis for 
proposed changes in the level 
of appropriations (Article 38(3)
(e)(ii), and a summary of evalu‑
ation results when relevant to 
budget changes (Article 38(3)
(e)(v)).

(iv)  The authorising officer by del‑
egation must report to his or 
her institution on the perfor‑
mance of his or her duties in 
the form of an annual activity 
report containing financial and 
management information and 
indicating the results of the 
operations by reference to the 
objectives set, the risks associ‑
ated with those operations, 
the use made of the resources 
provided and the efficiency 
and effectiveness of internal 
control systems (Article 66(9)).

(v)  The Court of Auditors’ special 

reports must be transmitted 
without delay to the European 
Parliament and the Council 
(Article 163).

2. Results policy and regulatory background

12

20  Regulation (EU) No 236/2014 
of the European Parliament 
and of the Council of 
11 March 2014 laying down 
common rules and procedures 
for the implementation of the 
Union’s instruments for 
financing external action (OJ 
L 77, 15.3.2014, p. 95).

(c)  The regulation on common rules 

(iv)  From 2015, the Commission 

must submit to the European 
Parliament and to the Coun‑
cil an annual report on the 
achievement of the objectives 
of each regulation by means 
of indicators, measuring the 
results delivered and the ef‑
ficiency of the relevant instru‑
ment; this report must assess 
the results of the Union’s 
financial assistance using, as 
far as possible, specific and 
measurable indicators of its 
role in meeting the objectives 
of the instruments (Article 13).

and procedures for the implemen‑
tation of the Union’s instruments 
for financing external action stipu‑
lates the following20.

(i)  Financing decisions must 

include in an annex a descrip‑
tion of each action, specifying 
its objectives, main activities, 
expected results, methods 
of implementation, budget 
and indicative timetable 
and performance monitor‑
ing arrangements (preamble, 
paragraph 5).

(ii)  The Union’s external action 
under the Instruments must 
contribute to clear results 
(covering outputs, outcomes 
and impacts); the results of the 
Union’s external action and 
the efficiency of a particular 
Instrument must be monitored 
and assessed on the basis of 
predefined, clear, transpar‑
ent and, where appropriate, 
country‑specific and measur‑
able indicators (preamble, 
paragraph 12).

(iii)  Action programmes must 

specify for each action the ob‑
jectives pursued, the expected 
results and the main activities 
(Article 2).

3. Our review

13

22 
We carried out our review work 
between October 2014 and January 
2015. It consisted of a desk review of 
documentation from EU institutions 
and other major multilateral and bilat‑
eral aid donors, as well as interviews 
with EU Commission staff, including 
from the EU delegation in Cambodia, 
and officials from two international 
organisations and several implement‑
ing bodies. A screening of ECA special 
reports published in recent years was 
also carried out in order to illustrate 
the key risk areas identified with 
specific examples. The timescale of 
development actions is such that these 
examples sometimes relate to actions 
undertaken several years ago.

20 
Our review aimed to provide in a sin‑
gle document an overview of what 
the Court considers to be the main 
risk areas related to a results approach 
for EU development and coopera‑
tion action. It also illustrates the risks 
concerned on the basis of experience 
acquired by the Court through its au‑
dits in the field of external actions. It is 
particularly relevant in the context of 
current discussions related to the Com‑
mission’s performance framework (see 
paragraph 5) and the implementation 
of a results framework for EU interna‑
tional cooperation and development 
action (see paragraph 11).

21 
Our review was guided by the follow‑
ing question: did the Commission ad‑
equately address the key risks entailed 
by a results approach for EU develop‑
ment and cooperation measures?

4. Key risk areas

14

23 
On the basis of its review work, the 
Court identified nine key risk areas, 
which are detailed and illustrated be‑
low. They relate to:

(a)  the consistent use of results re‑
lated terminology (section 4.1);

(b)  the establishment of a clear results 

chain (section 4.2);

(c) 

increased complexity due to the 
integration of cross‑cutting issues 
(section 4.3);

(d)  the harmonisation of instruments 
and frameworks between develop‑
ment partners (section 4.4);

(e)  results reporting and evaluation 

(section 4.5);

(f)  data consolidation and results 

overview (section 4.6);

(g)  data quality (section 4.7);

(h)  focus on budgetary outturn (sec‑

tion 4.8); and

(i)  changes in the context of EU ac‑

tions (section 4.9).

4.1. Formulation of 
objectives and indicators

4.1.1. Terminology

24 
Given the number and diversity of 
parties involved in, or associated with, 
implementing a new framework for 
EU development and cooperation, it is 
crucial for them to be able to employ 
a common and consistent terminology.

25 
However, the Court found that re‑
sults‑related terminology is not always 
used consistently (see Box 1).

26 
A lack of clarity and consistency in 
terminology employed can cause mis‑
understanding and confusion among 
those involved in EU development and 
cooperation activities, notably in the 
context of evaluating actions.

4. Key risk areas

15

1
x
o
B

Examples of terminology issues

While the regulation on common rules and procedures for the implementation of the Union’s instruments 
for financing external action21 provides that results cover outputs, outcomes and impact (see paragraph 19), 
EuropeAid’s management plan for 2014 uses a different categorisation in which outcomes include results and 
impact22.

While regulations provide for specific, measurable, achievable, relevant and timed (SMART) objectives, Euro‑
peAid’s Results Oriented Monitoring Handbook, which constitutes a reference document for all EuropeAid’s 
results‑oriented monitoring activities, refers to ‘SMART OVIs’, where ‘OVIs’ stands for objectively verifiable 
indicators; it thus confuses objectives and indicators; similarly, the Commission’s budget circular for 2014 
referred to SMART as the ‘minimum conditions to be fulfilled by objectives and indicators’23.

While regulations provide for predefined, clear, transparent and — where appropriate — country‑specific and 
measurable indicators, other sets of quality criteria are sometimes used, as EuropeAid pointed out in its 2012 
guidelines for budget support24: ‘Different sets of criteria exist to assess indicator quality: SMART indicators 
are Specific, Measurable, Achievable, Realistic and Time‑limited. CREAM criteria refer to indicators that are 
Clear, Relevant, Economic, Adequate and Monitorable. Eurostat indicator guidance focuses on logic, relevance, 
the possibility of setting a target, frequency of data collection, appropriateness and possibility of estimating 
precision. RACER indicators refer to indicators that are Relevant, Accepted, Credible, Easy, and Robust.’

21  Regulation (EU) No 236/2014.

22  See, for example, the DCI (Development Cooperation Instrument) intervention logic in section 4.1.4 of EuropeAid’s management plan for 2014.

23  European Commission — Budget circular for 2014 — Standing instructions, 14.12.2012.

24  Budget Support Guidelines — Programming, Design and Management — A modern approach to Budget Support, September 2012.

4.1.2. Document alignment 
and consistency

27 
Objectives — together with their indi‑
cators — are easier to understand and 
more likely to be achieved when they 
are formulated in a clear and consist‑
ent way in all interrelated documents.

 
16

25  OJ L 77 2014, 15.3.2014.

26  A logical framework is 

a document that provides an 
overview of a project’s 
objectives, activities and 
resources. It also usually 
outlines critical assumptions 
on which the project rests and 
provides information on how 
the project will be monitored, 
including through the use of 
indicators.

4. Key risk areas

28 
Objectives and indicators for EU devel‑
opment and cooperation are specified 
in a broad variety of documents. These 
include:

(a)  the Treaty on the Functioning of 

the European Union;

(b)  high‑level development goals (see 

section 2.1.1);

(c)  regulations on instruments for 
financing external action25;

(d)  the Commission’s regional and 
country strategies and multian‑
nual indicative programming 
documents;

(e)  annual action programmes an‑

nexed to the Commission’s financ‑
ing decisions;

(f)  financing agreements with partner 
countries, administration/contribu‑
tion agreements with international 
organisations, grant agreements 
and contracts with non‑govern‑
mental organisations and service 
providers;

(g)  activity statements (or programme 
statements) as part of the budget‑
ary procedure;

(h)  annual management plans and 
annual activity reports of the 
Commission directorates‑general 
concerned.

29 
The administration of all these docu‑
ments, which originate from different 
authorities and are subject to differ‑
ent adoption cycles and procedures, 
necessarily leads to adjustments to 
objectives and indicators and, in some 
cases, inconsistencies. In this connec‑
tion, a results reporting pilot exercise 
carried out by the Commission in 
2014 in the preparation of the results 
framework found that, where logical 
frameworks existed at both decision 
and contract levels, ‘the degree of co‑
herence between them was mixed — 
at best’26.

30 
In previous audits, the Court also 
found inconsistencies between objec‑
tive and indicator‑setting documents 
(see Box 2).

2
x
o
B

Example of inconsistencies between documents

‘The annual activity reports contain seven indicators for Central Asia. Though of general relevance to the suc‑
cess of EU development assistance in Central Asia, these indicators are different from those set in the Regional 
Strategy Paper and Multi‑annual indicative programmes; nor are they directly linked to the intended results of 
the Council’s new partnership strategy for Central Asia.’27

27  Special Report No 13/2013 ‘EU development assistance to Central Asia’, paragraph 79 (http://eca.europa.eu).

 
17

28 

In April 2014, the European 
Parliament urged the 
adoption of specific, 
measurable, achievable, 
relevant and timed 
benchmarks for all 
programmes in Heading 4 
— Global Europe — of the 
multiannual financial 
framework 2014‑2020. See the 
European Parliament Decision 
of 3 April 2014 on discharge in 
respect of the implementation 
of the general budget of the 
European Union for the 
financial year 2012, Section 
III — Commission and 
executive agencies — ref. 
P7_TA‑PROV(2014)0287.

4. Key risk areas

31 
Inconsistencies in the formulation of 
objectives and indicators can give rise 
to confusion about what should serve 
as a basis for assessing what has been 
achieved.

4.1.3. Setting SMART 
objectives and clear 
indicators

32 
Setting specific, measurable, achiev‑
able, relevant and time bound (SMART) 
objectives is a legal requirement for 
all EU development and cooperation 
activities. Moreover, the results of EU 
external action — including EU devel‑
opment and cooperation — and the 
efficiency of a particular instrument 
should be monitored and assessed on 
the basis of predefined, clear, transpar‑
ent and — where appropriate — coun‑
try‑specific and measurable indicators 
(see paragraph 19)28.

2 Need for SMART objectives and clear indicators
e
r
u
g
i
F

SMART objectives

Clear indicators

Input

Activities

Outputs

Outcome

Impact

s
t
l
u
s
e
R

 
4. Key risk areas

18

33 
In previous audits, the Court often 
found that the objectives set were 
too vague or unrealistic (see Box 3). 
Similarly, the Court often found that 
indicators were missing or unclear 
(see Box 4). Even when indicators are 
well defined, their interpretation can 
be relative and their value can vary 
considerably according to the context 
in which they are used (see Box 5).

35 
As well as contravening regulations, 
failing to set SMART objectives and 
clear indicators undermines perfor‑
mance monitoring and results assess‑
ment, which is detrimental to account‑
ability and public audit. The results of 
an intervention also risk being limited 
if its objectives are not well defined 
from the beginning.

34 
The results reporting pilot exercise car‑
ried out by the Commission in 2014 in 
the preparation of the results frame‑
work also found that, in many cases, 
where the logical frameworks existed 
in decisions and contracts, they were 
not of good quality, i.e. objectives 
were not SMART and indicators were 
not well defined or were based on 
unclear data sources.

3
x
o
B

Example of objectives that were too vague or unrealistic

‘The lack of clear definition of project objectives relates mainly to equipment planned, its level of operation 
(quantity and quality) and the target population expected to benefit from project results. The lack of clear and 
quantified measures significantly hampered the Commission’s ability to monitor progress during implementa‑
tion and later to measure and check whether the results achieved were sustained over time or not (...).’29

‘The objectives are also sometimes overly ambitious, in particular in the case of NGO projects which have 
modest budgets and a relatively short implementation period. This was particularly true of NGO projects 
funded under the Food Facility: they had approximately 20 months for implementation, which in practice was 
reduced to more like 15 months as the first months were given over to procurement and other preparatory 
activities. In some cases, the interventions are based on unreasonable assumptions concerning the conditions 
necessary for successful implementation, e.g. the institutional and management capacities of the national or 
local authorities, the quality of rural infrastructure or the availability of suitable staff (e.g. nutritionists and 
extension workers).’30

29  Special Report No 13/2012 ‘European Union development assistance for drinking water supply and basic sanitation in sub‑Saharan countries’, 

paragraph 50 (http://eca.europa.eu).

30  Special Report No 1/2012 ‘Effectiveness of European Union development aid for food security in sub‑Saharan Africa’, paragraph 52 (http://eca.

europa.eu).

 
4. Key risk areas

19

4
x
o
B

5
x
o
B

Examples of indicators that were missing or unclear

‘No performance indicators were included in the financing agreements for Pegase Direct Financial Sup‑
port (Pegase DFS), which makes it harder for the EEAS, the Commission and the Member States to assess the 
concrete results of the support (...). The lack of performance indicators also makes it more difficult to dem‑
onstrate results in order to attract new funding. Moreover, this does not comply with the ENPI regulation’s 
explicit requirement that performance indicators be established for monitoring purposes for all forms of ENPI 
assistance.’31

‘(…) in Kyrgyzstan the food security information programme contained vague indicators, such as ‘food secu‑
rity information is generally accepted to be accurate and timely’, ‘adequate numbers of staff are trained’ and 
‘usefulness of crop forecast’.’32

31  Special Report No 14/2013 ‘European Union direct financial support to the Palestinian Authority’, paragraph 31 (http://eca.europa.eu).

32  Special Report No 13/2013, paragraph 52.

Example of the changing relevance of indicators

The EU provides budget support to Cambodia for the development of its education sector. In recent years, 
grade 12 school exams in Cambodia were undermined by cheating and corruption. The pass rates were then 
around 80 %. In 2014, the Cambodian authorities undertook to prevent and eventually eliminate irregulari‑
ties in these exams, in particular by introducing measures at testing centres to prevent cheating. A first exam 
organised in August 2014 led to a 25 % pass rate; a second‑chance exam organised in October brought the 
total pass rate for 2014 up to 41 % of candidates, roughly half the 2013 pass rate. At first sight, the change in 
this indicator could be interpreted as a deterioration; however, it can alternatively be regarded as reflecting an 
improvement in the reliability and credibility of the exams.

4.1.4. Defining a partner 
country’s commitment to 
implement reforms as an 
objective

36 
For an action’s objectives to be sig‑
nificant in terms of development, they 
should correspond to the achievement 
of a result, i.e. a measurable change 
resulting from the action concerned.

 
 
4. Key risk areas

20

38 
The effectiveness of EU development 
aid and coordination actions can be 
undermined when the partner coun‑
try’s apparent commitment to achiev‑
ing results, without further evidence 
that significant reforms are actually 
being implemented, is considered as 
a sufficient condition for payment.

37 
In previous audits, the Court found 
that the objectives of EU actions are 
sometimes formulated in the form of 
a commitment to be taken by a part‑
ner country at a given moment. How‑
ever, in some of these cases, even if 
the stated objective had been met, i.e. 
even if the partner country appeared 
to have made the expected commit‑
ment, the action had then no or little 
significant effect over time; in other 
words, it did not produce any signifi‑
cant result in terms of development 
(see Box 6).

6
x
o
B

Example of intervention where the partner country’s commitment to implement 
reforms did not translate into development results

‘The Commission disbursed its budget support in Kyrgyzstan in three tranches, each of which was preceded 
by an assessment of progress towards public financial management (PFM) objectives. The Commission as‑
sessed that progress had been slow, but justified the disbursement of each of the three tranches mainly 
because it considered that the Kyrgyz government remained committed to further reforms. There was no 
clear evidence for this opinion. A core part of the Commission’s reasoning for its disbursement decisions is 
based on its assessment of the partner countries’ commitment to reform rather than on the actual progress 
achieved’.33

‘The Commission did not succeed in ensuring that the issues it brought up in the subcommittee dialogue 
were subsequently followed up by the Egyptian authorities with concrete actions. Even when issues raised 
were included as action items at the end of the meeting, they were not generally implemented. For example, 
the 2009 subcommittee included as an action point ‘exploring the possibility to use EU technical assistance to 
advise Egypt on NGO legislation in Member States’. This was not followed up. (…) The Commission made no 
link between its criticisms of human rights violations made in the progress reports and the option of reducing 
or suspending EU assistance.’ 34

33  Special Report No 13/2013, paragraph 57.

34  Special Report No 4/2013‘ EU Cooperation with Egypt in the Field of Governance’, paragraphs 39 and 41 (http://eca.europa.eu).

 
4. Key risk areas

4.2. Establishment of the 
results chain

4.2.1. Linking actions and 
results through a clear 
logical chain

39 
In order to obtain the expected results, 
it is necessary to identify appropriate 
actions and thus to establish a clear 
logical link between activities, outputs, 
outcome and impact35,36. Establishing 
such a logical chain is also essential 
to demonstrate how EU development 
and cooperation interventions contrib‑
ute to results37.

42 
In the absence of a clear link between 
actions — or conditions for pay‑
ment — and their intended results, 
the interventions’ objectives can be 
lost sight of, which is detrimental to 
aid effectiveness; the EU’s capacity to 
demonstrate and account for the re‑
sults of its actions is also undermined. 
Similarly, seemingly separate policies 
can in reality be closely linked40. When 
designing an action in the context of 
a given policy, its possible results in 
terms of other policies may be given 
insufficient consideration; for example, 
a project could contribute to health‑re‑
lated objectives, such as combating 
malaria, while its stated objectives 
only concern improving access to safe 
drinking water.

40 
Some previous Court special reports 
and Commission evaluation reports 
found that the link between action 
and results was not always clearly 
established (see Box 7).

41 
For some actions, results‑based pay‑
ment conditions can be set in financ‑
ing agreements linking payment di‑
rectly to the achievement of results in 
order to create incentives for improved 
performance through full or partial 
payment for full or partial achieve‑
ment of results38,39. In previous audits, 
the Court sometimes found that such 
a link was not made or enforced (see 
Box 8).

21

35  Study on Legal Instruments 

and Lessons Learned from the 
Evaluations Managed by the 
Joint Evaluation Unit — Final 
Report Volume 1: Final 
Summary Report, July 2011, 
p. vi: ’Formulating clear logic 
chains in regulations is key. 
This can be encouraged by 
avoiding unstructured lists of 
activities and unclear links 
between activities, outputs 
and their intended impacts. 
The relationships between 
each step in the chain should 
be clear and logical so as to 
avoid gaps in the logic and 
‘orphan’ outputs or 
impacts.‘ — ref. evaluation 
DEVCO 1292.

36  A detailed explanation of 

a results chain is provided in 
United Nations Development 
Group’s results‑based 
management handbook, 
October 2011, p. 14.

37  Study on the uptake of 

learning from EuropeAid’s 
strategic evaluations into 
development policy and 
practice, Final report, 
June 2014, p. vii.

38  Commission Budget Support 
Guidelines — Executive 
Guide — A modern approach 
to Budget support, 
September 2012.

39  See EU Common Position for 
the Fourth High Level Forum 
on Aid Effectiveness (Busan, 
29 November — 
1 December 2011) — Council 
conclusions — 3124th Foreign 
Affairs Development Council 
meeting, Brussels, 
14 November 2011: ‘The EU 
calls on partner countries and 
development partners to 
increase the emphasis on 
harmonised and results‑based 
conditionality’.

40  Thematic evaluation of the 
European Commission 
support to the health 
sector — Final report — 
Volume 1, August 2012, p. xiv.

4. Key risk areas

22

7
x
o
B

8
x
o
B

Examples of an unclear link between action and results

‘In the 24 interventions examined, the ENP strategy is clearly referenced as the umbrella under which the 
cooperation takes place. Reference to one or several ENP objectives is clearly made in the programming docu‑
ments and the interventions are in line with the strategy documents. However, the documentation reviewed 
does not show nor explain in an explicit manner how the objectives will be achieved through the specific 
interventions, and how the interventions programmed will contribute to achieve the results.’41

‘Budget support programmes in Malawi and Rwanda make reference to the objectives of the national poverty 
reduction strategies and sector policies and programmes. They do not state how they aim to contribute to the 
governments’ objectives.’42

41  Evaluation of the European Union’s Support to two European Neighbourhood Policy Regions (East and South) — Final report — Volume 1, June 

2013, p. 24.

42  Special Report No 1/2012, paragraph 53.

Examples of financial interventions with ineffective payment conditions

‘One objective of the two targeted budget support programmes was to contribute to improving public finan‑
cial management (PFM). However, the programmes did not establish any links with the government’s PFM re‑
form plan, nor did they identify which priorities they were intended to support. It is unclear how programmes 
of this sort can achieve such an objective, given that funds were disbursed in a single fixed tranche, with no 
performance‑related conditions or policy dialogue requirements.’43

‘For road infrastructure investment projects, the Commission uses conditions in a way that gives partner coun‑
tries little incentive to comply with them. The conditions set are not legally binding but presented in the fi‑
nancing agreements as ‘accompanying measures’ to be taken up by the partner country. In practice, the Com‑
mission generally endorses works contracts between partner countries and contractors regardless of whether 
these accompanying measures have been taken. It is then under an obligation to pay EDF money under these 
contracts, and it has little leverage to compel the partner country to comply with the conditions.’44

43  Special Report No 9/2013 ‘EU support for governance in the Democratic Republic of the Congo’, paragraph 65 (http://eca.europa.eu).

44  Special Report No 17/2012 ‘The European Development Fund (EDF) contribution to a sustainable road network in sub‑Saharan Africa’, 

paragraph 27 (http://eca.europa.eu).

 
 
23

45  OECD — DAC Network on 

Development Evaluation — 
Evaluating Budget Support 
Methodological Approach, 
page 11. See also page 12, on 
budget support outcomes: 
‘Outcomes are the result of the 
entire set of policies, strategies 
and spending actions of the 
government (…), of the 
reactions/responses of the 
relevant stakeholders to 
changes in public policy 
making and resource 
allocation decisions, and of 
the influence of other external 
factors. These outcomes are 
thus only partly influenced by 
the BS provided. The causal 
relationship between the 
provided budget support and 
the outcomes will therefore be 
(strongly) diluted with other 
influencing factors.’

4. Key risk areas

4.2.2. Contribution versus 
attribution

43 
EU development and cooperation 
interventions take place in a complex 
realm, where many other donors and 
many exogenous factors, besides EU 
intervention, contribute — positively 
or negatively — to the achievement of 
results. For this reason, as well as es‑
tablishing a causal link between EU de‑
velopment and cooperation action and 
its results (see section 4.2.1.), it could 
be desirable to assess the strength of 
this link, i.e. which part of the results 
can be attributed to EU action.

44 
However, as the Commission pointed 
out in its Staff Working Document on 
‘Paving the way for an EU develop‑
ment and Cooperation Results Frame‑
work’ (see paragraph 7), ‘it is difficult 
to measure contribution of one level 
of the results hierarchy to the next’, 
i.e. from output to outcome, and from 
outcome to impact. Indeed, while an 
intervention’s outputs can usually be 
largely attributed to the intervention, 
this will often not be the case for its 
outcome, and even less for its impact, 
as the Court found in some of its au‑
dits (see Box 9).

45 
In the particular case of budget sup‑
port interventions, it is not normally 
possible to identify the outputs, due to 
the fungibility of EU financial support 
within the partner country’s budget. 
Indeed, the partner country’s budget 
supports the whole country’s develop‑
ment strategy, and the outputs of EU 
budget support interventions thus 
relate to changes of government poli‑
cies and services, which are not the 
consequence of the budget support 
programme alone but also of various 
other government actions and deci‑
sions, other external assistance pro‑
grammes (…) and external non‑gov‑
ernment‑related factors45. This is well 
illustrated in the OECD’s comprehen‑
sive evaluation framework for budget 
support evaluations (see Annex).

46 
Consequently, even where a clear logi‑
cal link between action and results can 
be established, it can be difficult — if 
not impossible — to assess which part 
of the results is attributable to EU in‑
terventions and which is due to other 
donors’ interventions or other factors. 
For this reason, it is more relevant to 
speak in terms of contribution rather 
than attribution. Besides its techni‑
cal aspects, this issue also relates to 
the more political question of results 
ownership. Indeed, when many stake‑
holders are involved, including the EU, 
partner countries and other donors, 
there can be several claims for the 
same positive results, or no acknowl‑
edgement of responsibility for nega‑
tive results.

4. Key risk areas

24

9
x
o
B

Example of difficulties in attributing results to EU intervention

‘The Court cannot assess the extent to which the EU budget support programmes in Malawi and Rwanda have 
contributed to the results of the governments’ programmes since there are multiple and complex intervening 
factors (…).’46

47  EEAS and Commission joint 

‘Instructions for the 
programming of the 11th 
European Development Fund 
(EDF) and the Development 
Cooperation Instrument 
(DCI) — 2014‑2020’.

48  See also, for example, the 

Development and 
Cooperation Instrument (DCI) 
Regulation (EU) No 233/2014 
which lays down that: 
‘Throughout all programmes, 
cross‑cutting issues as defined 
in the European Consensus 
shall be mainstreamed. In 
addition, conflict prevention, 
decent work and climate 
change shall be 
mainstreamed, where 
relevant’ (Article 3).

46  Special Report No 1/2012, paragraph 57.

4.3. Increased complexity 
due to the integration of 
cross‑cutting issues

47 
In its communication on ‘Increasing 
the impact of EU Development Policy: 
an Agenda for Change’ (see para‑
graph 18), the Commission proposed 
that EU activities in each partner 
country should be concentrated on 
a maximum of three sectors. Focusing 
EU interventions on a small number of 
sectors can increase their impact and 
leverage and help to keep EU assis‑
tance manageable not only for partner 
countries, but also for the Commission 
and EU Delegations47.

48 
Meanwhile, according to EU instruc‑
tions for establishing multiannual 
programming documents for the 
2014‑2020 programming period, sector 
policy commitments must also take 
into account nine cross‑cutting issues, 
including the promotion of human 
rights, gender equality, democracy, 
good governance, children’s rights, 
persons with disabilities, indigenous 
peoples, environmental sustainabil‑
ity and combating HIV/AIDS48 (see 
Box 10).

 
4. Key risk areas

25

3 Focal sectors and cross‑cutting issues
e
r
u
g
i
F

Focal sector 1

Focal sector 2

Focal sector 3

Cross-cutting issue 1

Cross-cutting issue 2

Cross-cutting issue 3

Cross-cutting issue 4

Cross-cutting issue 5

Cross-cutting issue 6

Cross-cutting issue 7

Cross-cutting issue 8

Cross-cutting issue 9

49 
Previous Commission evaluation 
reports pointed out difficulties in 
mainstreaming human rights through 
EU development actions (see Box 11).

50 
The mainstreaming of cross‑cutting 
issues in EU interventions can make 
the prioritisation of these interven‑
tions’ objectives more complex. The 
increased complexity due to the 
integration of cross‑cutting issues in 
EU programmes represents a supple‑
mentary risk to the achievement and 
reporting of results.

0
1
x
o
B

Integration of cross‑cutting issues in a programme — the example of Cambodia

The Commission’s Multiannual Indicative Programme for Cambodia for the 2014‑2020 period provides for 
three focal sectors: Agriculture/Natural Resource Management, Education/Skills and Good Governance. It also 
provides for the mainstreaming of several cross‑cutting issues: gender equality and empowerment of women, 
nutrition, the environment, and information and communication technologies.

 
 
4. Key risk areas

26

1
1
x
o
B

Examples of difficulties in addressing cross‑cutting issues — the case of human 
rights

‘The EU has sought to further translate its commitment and mainstream human rights at programming level, 
including regional strategy papers, bilateral action plans, and thematic programmes. Human rights and de‑
mocracy are entrenched in the policy documents as guiding principles. Nevertheless, one can observe a phe‑
nomenon of ‘dilution’ of the democracy and human rights component as the cycle moves ‘downstream’, i.e. 
from broad political pledges to policy frameworks down to programming, choice of instruments, allocation of 
funding and selection of projects. This ‘dilution’ appears under various forms including: (i) the lack of coherent 
strategy to address problems through concrete interventions; (ii) the limited funding for related democratic 
and human rights aspects in other thematic important sectors; (iii) the tendency to confine human rights work 
to thematic instruments.’49

‘The desk analysis of six concrete areas where European Commission (EC) sought to mainstream human rights 
shows the phenomenon of ‘dilution’ of the human rights component as the cycle moves downstream (…). 
All this is compounded by a lack of even basic systems to monitor and evaluate progress in relation to main‑
streaming. This dilution is related to political resistance of partner countries but also to major internal weak‑
nesses at EU level, such as limited political leverage and inconsistent decision‑making with regard to the place 
and weight of human rights in EU external action. The EU’s difficulties to reconcile values and interests have 
been widely documented and criticised by media and civil society. The Arab Spring was an eye‑opener on the 
limits of the ‘stability versus human rights approach’ and may open perspectives for a more serious approach 
to mainstreaming human rights.’ 50

49 

‘Evaluation of the European Union’s Support to two European Neighbourhood Policy Regions (East and South) — Final report — Volume 1, June 
2013’, p. 62.

50 

‘Thematic evaluation of the European Commission support to respect of Human Rights and Fundamental Freedoms (including solidarity with 
victims of repression)’ — Final Report — Volume 1, December 2011, p. 15.

 
27

51  Busan Partnership for effective 
development cooperation.

4. Key risk areas

4.4. Harmonisation 
between development 
partners’ instruments and 
frameworks

4.4.1. Alignment with 
partner countries’ results 
frameworks

51 
In line with the principles agreed 
at high‑level international forums, 
donors, including the EU, should rely 
as far as possible on partner coun‑
tries’ frameworks for results‑oriented 
reporting and monitoring (see sec‑
tion 2.1.2). In particular, donors should 
minimise their use of additional frame‑
works, refraining from requesting the 
introduction of performance indicators 
that are not consistent with countries’ 
national development strategies51.

52 
However, previous Commission evalu‑
ation reports found issues related to 
non‑alignment between donors and 
partner countries’ frameworks (see 
Box 12).

53 
When partner countries face many 
different types and levels of reporting 
obligations and payment conditions 
from various donors, these can present 
an unmanageable burden, which is 
detrimental to both aid efficiency and 
effectiveness.

2
1
x
o
B

Example of the lack of alignment with the partner country’s results framework

‘(…) far from being a Government‑led, genuinely harmonised framework, the Performance Assessment Frame‑
work (PAF) had developed into ‘the sum total of all Development Partners’ individual preferences and require‑
ments’. (…) The effective ownership of Government was lost and the PAF assessment process became little 
more than a compulsory process, which needed to be respected in order for Budget Support disbursements 
to continue (…). Simultaneous with this loss of ownership and engagement, there has been a significant in‑
crease in the transaction costs associated with the annual assessment process and the related policy dialogue. 
This is a perception reported unanimously by all relevant stakeholders interviewed. It is manifested most obvi‑
ously in the steady growth in the number of indicators incorporated within the PAF.’52

52  Joint Evaluation of Budget Support to Tanzania: lessons learned and recommendations for the future Final Report: Volume 1, 2013, 

paragraphs 74‑75.

 
4. Key risk areas

28

4.4.2. Harmonising the 
instruments used by 
development stakeholders

54 
In previous Court special reports, is‑
sues were found related to the lack of 
harmonisation of instruments used by 
development partners (see Box 13).

55 
The failure to harmonise instruments 
used by development partners can be 
detrimental to results monitoring and 
reporting effectiveness and efficiency.

3
1
x
o
B

Examples of issues due to the lack of harmonisation of instruments used by 
development partners

‘The United Nations Development Programme (UNDP) has estimated that there are more than 50 international 
public funds, 45 carbon markets and 6 000 private equity funds providing climate change finance, each with 
its own governance structure. As the World Bank notes, fragmentation of this sort threatens to reduce the 
overall effectiveness of climate finance (…). The Commission and Member States use both bilateral and multi‑
lateral channels to disburse climate finance. In 2010 they used no less than 22 multilateral channels.

(…) No attempt has been made to reduce the proliferation of climate funds, which involves serious risks of 
inefficiencies, inadequate accountability and fragmentation of aid (…)’53

53  Special Report No 17/2013 ‘EU climate finance in the context of external aid’, paragraphs 57 to 58 and paragraph 68 (http://eca.europa.eu).

57 
In previous audits, the Court found 
that the juxtaposition of different ac‑
countability structures for tasks under 
indirect management with internation‑
al organisations sometimes created 
accountability gaps (see Box 14).

54  Financial Regulation, 

Article 58 — methods of 
implementation of the 
budget.

55  See, for example, United 

Nations Development Group’s 
results‑based management 
handbook, United Nations 
Population Fund’s integrated 
results framework, or United 
Nations Development 
Programme’s handbook on 
planning, monitoring and 
evaluating for development 
results.

4.4.3. Coping with the 
juxtaposition of different 
accountability structures

56 
As part of the EU’s development and 
cooperation action, the European 
Commission entrusts some budget 
implementation tasks under indirect 
management to international organi‑
sations54. For these tasks, accountabil‑
ity relies on the accountability struc‑
tures of the international organisation 
concerned, including governance, 
results frameworks55 and external 
audit, as well as on the EU’s own audit 
and accountability provisions.

 
4. Key risk areas

29

4
1
x
o
B

Example of distinct accountability architecture — United Nations

‘Although the Commission verifies the financial management systems of its UN partners (using a four‑pillar 
analysis), the Commission depends on UN reports for confirming the practical operation of control systems 
and the achievement of results. At the time of the audit the Commission had not yet succeeded in obtaining 
adequate information from the UN in this regard.’56

‘The Commission does not receive sufficient timely information from UN reports. A large proportion of reports 
are still delayed, not detailed enough and focus on activities rather than results.’57

56  Landscape review 2014 ‘Gaps, overlaps and challenges: a landscape review of accountability and public audit arrangements’, Box 1 (http://eca.

europa.eu).

57  Special Report No 3/2011 ‘The efficiency and effectiveness of EU contributions channelled through United Nations organisations in 

conflict‑affected countries’, paragraph IV (http://eca.europa.eu).

4.5. Weaknesses in 
evaluation and results 
reporting

58 
Legal provisions for results monitoring, 
evaluation and audit provide for two 
different types of timing:

(a)  periodic timing: for example, 

Article 318 TFEU lays down that the 
Commission must submit annu‑
ally (…) an evaluation report on 
the Union’s finances based on the 
results achieved, in particular in 
relation to the indications given 
by the European Parliament and 
the Council in the context of the 
annual discharge procedure;

(b)  action‑based timing: for example, 
Article 18.3 of the rules of applica‑
tion of the Financial Regulation 
provides that results of multian‑
nual programmes where the 
resources exceed 5 000 000 euros 
must be periodically evaluated in 
accordance with a timetable which 
enables the findings of that evalu‑
ation to be taken into account 
for any decision on the renewal, 
modification or suspension of the 
programme.

59 
In previous audit reports, the Court 
found that, in some cases, the achieve‑
ment of projects or programme objec‑
tives could not be assessed before 
they finished, which undermined peri‑
odic assessment of the achievement of 
results (see Box 15).

 
4. Key risk areas

30

61 
Weaknesses in results reporting and 
evaluation timing can prevent the 
feedback of relevant knowledge and 
lessons‑learnt into reporting and 
decision‑making processes.

58  Special Report No 18/2014 

‘EuropeAid’s evaluation and 
results‑oriented monitoring 
systems’, paragraphs 27 to 29 
(http://eca.europa.eu).

59  Study on the uptake of 

learning from EuropeAid’s 
strategic evaluations into 
development policy and 
practice — Final report, June 
2014, p. vii.

60 
The Court also found weaknesses in 
the timely delivery of evaluations, i.e. 
strategic evaluations often took much 
longer than planned and programme 
evaluations were sometimes post‑
poned or cancelled58. A recent study 
requested by the Commission also 
found ‘important disconnects’ be‑
tween evaluations and policy for‑
mulation, programming, monitoring 
and result‑oriented management. In 
particular, it found missed opportuni‑
ties — in terms of uptake — where 
previously learned lessons were 
ignored because evaluation findings 
did not synchronise with the manage‑
ment decision‑making process, (…) 
or because the context had changed 
by the time the findings were made 
available59.

4 The time cycle of an action
e
r
u
g
i
F

Legislative
process

Follow-up
(including evaluation)

Programming

Implementation
(including monitoring and
mid-term evaluation)

Identiﬁcation

Formulation

 
4. Key risk areas

31

5
1
x
o
B

Example of results which could only be assessed at the end of the action concerned

‘The Court assessed the effectiveness of a sample of 19 projects by comparing the intended objectives in the 
contribution agreement with the actual achievements. (…) Due to insufficient information, the results could 
not be assessed in the following cases: (…) One major project was a trust fund for which the objectives stated 
in the contribution agreement concerned the fund as a whole and were therefore of a more general nature. 
Although more specific objectives were defined for the underlying projects, there can be no clear link be‑
tween the achievement of the project objectives and the general objectives in the contribution agreement. 
Consequently the effectiveness can only be assessed when the fund itself will be closed, which was not yet 
the case at the time of the audit.’60

60  Special Report No 3/2011, paragraphs 25 and 27 (b).

4.6. Data consolidation 
and reporting

62 
In order to demonstrate the results 
achieved by EU development poli‑
cies, data on results achieved by EU 
development and cooperation actions 
needs to be collected and consolidat‑
ed through information systems and 
presented and disseminated through 
appropriate reporting channels.

63 
However, in previous audit reports, the 
Court found issues with the capacity 
of EuropeAid’s information systems 
to collect results‑related data (see 
Box 16). In 2014, the Commission also 
pointed out that, for the EU to be 
able to report results, data collection 
systems and tools would need to be 
further rationalised and developed61.

64 
Data on EU development aid and 
cooperation activities serve as a basis 
for a broad variety of reports, the main 
ones being:

(a)  the Commission’s annual accounts 
and report on budgetary and 
financial management62;

(b)  EuropeAid’s Annual Activity Report 
and the Commission’s synthesis 
report63;

(c)  annual reporting to the OECD’s 
Development Assistance Com‑
mittee (DAC), in line with DAC’s 
directives64;

(d)  a monthly data contribution to the 
IATI Data Registry, in line with IATI 
standards65;

(e)  the annual report on the evalua‑

tion of the Union’s finances based 
on the results achieved66;

61  SWD(2013) 530 final, p. 12.

62  Financial Regulation, in 

particular Articles 141 to 142.

63  Financial Regulation, in 
particular Article 66.9.

64 

‘Converged statistical 
reporting directives for the 
creditor reporting system 
(CRS) and the annual DAC 
questionnaire’ and 
addendums — reference 
DCD/DAC(2013)15/FINAL, 
11.6.2013.

65 

Internet site of the 
International Aid Transparency 
Initiative (www.iatiregistry.
org).

66  Treaty on the Functioning of 

the European Union, 
Article 318.

 
4. Key risk areas

(f)  the annual report on the European 
Union’s development and external 
assistance policies and implemen‑
tation thereof;

(g)  regular updates to the Commis‑

sion’s ‘EU Aid Explorer’ website67.

65 
Currently most of these reports focus 
on activities, while none of them 
provides consolidated data on the 
results achieved by these activities 
across the board68. A lack of consoli‑
dated reports on and overview of the 
results achieved by EU aid can impair 
decision‑making and accountability.

32

67  http://tr‑aid.jrc.ec.europa.eu

68 

In particular, the annual report 
on the evaluation of the 
Union’s finances based on the 
results achieved does not 
provide a clear view of the 
extent to which the Union’s 
main objectives have been 
achieved. See Report on the 
evaluation of the Union’s 
finances based on the results 
achieved: a new tool for the 
European Commission’s 
improved discharge 
procedure — A7‑0068/2014, 
30.1.2014.

6
1
x
o
B

Example of weaknesses in EuropeAid’s information system

‘The main reporting tool used by the Commission between delegations and EuropeAid is the External As‑
sistance Monitoring Report (EAMR) (…). However, these reports largely describe activities carried out and 
implementation problems encountered. Moreover, information provided to a certain extent overlaps with 
information available in CRIS. They provide little indication on the actual results of the aid, either in terms of 
assessments made by the delegation itself or by reporting on the results of evaluations made after aid inter‑
ventions have closed. (...)’69

‘(…) CRIS cannot easily be used to compute the total amount spent on a given policy or financed from a given 
financial instrument. Indeed, the list of domains used to associate records from many CRIS modules to a given 
domain mixes geographical zones (Asia, for example), financial instruments (...) and thematic policies (food 
security, for example).

As a result of this situation, the consolidation of data — notably financial data — contained in CRIS for report‑
ing purposes is rendered particularly complex. This situation is detrimental to CRIS’s efficiency and effective‑
ness as a reporting and management tool.’70

69  Special Report No 1/2011 ‘Has the devolution of the Commission’s management of external assistance from its headquarters to its delegations 

led to improved aid delivery?’, paragraph 59 (http://eca.europa.eu).

70  Special Report No 5/2012 ‘The Common External Relations Information System (CRIS)’, paragraphs 40 and 41 (http://eca.europa.eu).

 
4. Key risk areas

33

4.7. Data quality

66 
Results monitoring and assessment 
requires the collection of sufficient, 
relevant, reliable and up‑to‑date data. 
In line with commitments made in 
high‑level forums on aid effective‑
ness (see section 2.1.2), data collection 
should rely as much as possible on 
partner countries’ own frameworks 
for results‑oriented reporting and 
monitoring.

67 
However, previous Court audits and 
Commission evaluations sometimes 
found that data sources did not 
provide sufficient, relevant, reliable 
and timely information on results (see 
Box 17). Data collection is sometimes 
costly — notably in the case of census‑
es — and thus not frequently carried 
out by partner countries.

68 
A lack of sufficient, relevant, reliable 
and up‑to‑date data can significantly 
weaken the quality and reliability of 
reported results.

7
1
x
o
B

Examples of data‑quality weaknesses

‘Generally the indicators used by the Commission have an appropriate focus on the [Millennium Development 
Goals], although insufficient attention is paid to indicators for education quality. However, the national educa‑
tion management information systems that the Commission relies on do not consistently provide sufficient, 
reliable and timely information.’71

71  Special Report No 12/2010 ‘EU development assistance for basic education in sub‑Saharan Africa and south Asia’, paragraph V (http://eca.europa.eu).

4.8. Budgetary outturn

69 
In previous years, there has been pres‑
sure to increase the amounts devoted 
to EU development and cooperation, 
particularly as part of the context of 
the commitment to attain the interna‑
tionally agreed ODA target of an ODA/
GNI ratio of 0.7 % by 201572.

72 

‘Member States undertake to 
achieve the target of an ODA/
GNI ratio of 0.7 % by 2015, 
while those which have 
achieved that target commit 
themselves to remaining 
above that target; Member 
States which joined the EU 
after 2002 will endeavour to 
increase their ODA/GNI ratio 
to 0.33 % by 2015’ — see 
Brussels European Council 16 
and 17 June 2005, Presidency 
conclusions — reference 
10255/1/05 REV 1, 
paragraph 27.

 
73 
Focusing on budgetary outturn as 
an objective can be detrimental to 
sound financial management and the 
achievement of results.

73 

34

‘One perennial question (…) is 
how to reconcile the 
long‑term nature of 
development cooperation, 
calling for multi‑year planning 
horizons, with the normal 
practice of aid appropriations 
lapsing each year. In many 
DAC member countries, 
general government 
procedures require that funds 
appropriated in a given fiscal 
year be disbursed within that 
year or, as a minimum, 
committed within that year 
and spent soon afterwards. As 
a consequence, aid managers 
in some countries operate 
under considerable pressure 
to commit and disburse funds 
rapidly, promoting undue 
emphasis on the financial 
inputs of development 
activities, rather than desired 
outcomes and actual results.’ 
— See OECD — Development 
Assistance Committee — 
Managing aid: practices of 
DAC Member countries — 
reference DCD/DAC(2004)40.

74  European Court of Auditors’ 

Annual report concerning the 
financial year 2013, 
paragraph 10.56 (OJ C 398, 
12.11.2014).

4. Key risk areas

70 
Article 317 TFEU stipulates that the 
Commission must implement the 
budget in accordance with the finan‑
cial regulations and taking into ac‑
count the principles of sound financial 
management. Article 38(3) of the 
Financial Regulation stipulates that the 
Commission must attach to the draft 
annual budget, among other things, 
information on the implementation 
rates for the preceding year’s activity 
and implementation rates for the cur‑
rent year.

71 
Budgetary outturn can be perceived as 
an objective in itself and is sometimes 
stated as such in Commission manage‑
ment plans (see Box 18).

72 
For this reason, there can be tension 
between the need to implement the 
budget and the need to implement 
it in accordance with the principle of 
sound financial management (effec‑
tiveness, efficiency and economy)73. In 
particular, achievements and failures 
in terms of budgetary implementa‑
tion are more directly visible than 
achievements in terms of performance 
(results). For this reason, there can be 
a natural tendency to focus on budg‑
etary implementation to the detri‑
ment of focusing on results, as the 
Court found in its annual report on the 
implementation of the budget in 2013: 
‘When spending EU funds in the 2007‑
2013 programming period, the focus 
was on absorption (the need to spend 
money) and compliance rather than 
good performance’74.

4. Key risk areas

35

8
1
x
o
B

Example of a budgetary execution objective

EuropeAid’s management plan for 2014 sets the following objective for the European Development Fund 
(EDF) in Africa75: ‘Levels of budgetary execution — Maintain high levels of budgetary execution including 
budget support payments’.

EuropeAid’s management plan also contains, for external financing instruments other than EDF, an annex 
entitled ‘Sound financial management and effective use of EC resources’, which sets several indicators and 
targets for budgetary execution:

Indicator

1

2

3

Execution of initial annual financial forecast: Payments

Execution of annual financial forecast for contracts

Execution of annual financial forecast for decisions

Target (2014)

From 90 % to 110 %

From 90 % to 110 %

From 90 % to 110 %

75  Management Plan 2014 — Directorate‑General for Development and Cooperation — EuropeAid.

4.9. Changes in EU 
actions’ context

4.9.1. Risk management and 
choice of implementation 
modalities

74 
EU development and cooperation 
actions are implemented in a large 
number of countries with different lev‑
els of development, many of which are 
characterised by critical social, political 
or humanitarian conditions or even 
volatile security situations. The choice 
of an aid implementation modality to 
achieve an action’s objectives should 
be based on its expected capacity 
to deliver results in an effective and 
efficient way given the context of the 
action. In some cases, the choice of 
an implementation modality that may 
appear riskier than others may some‑
times be motivated by the expectation 
that it will deliver better or quicker 
results.

 
36

76  European Court of Auditors’ 

Annual report concerning the 
financial year 2011, 
paragraph 7.10 (OJ C 344, 
12.11.2012).

77  COM(2011) 638 final of 

13 October 2011 ’The Future 
Approach to EU Budget 
Support to Third Countries‘.

76 
In previous audit reports, the Court 
sometimes found that the choice of an 
aid implementation modality had not 
been preceded by a suitable analysis 
of the risks entailed with regard to the 
expected advantages (see Box 19).

77 
Failing to compare the expected 
benefits with the potential disadvan‑
tages of the selected implementation 
modalities can lead to a wrong choice 
of modality or expose the EU budget 
to unreasonably high risks.

4. Key risk areas

75 
Budget support illustrates this situa‑
tion well. Indeed, while it can be con‑
ceived as a means of supplying large 
volumes of aid in a more predictable 
manner than the traditional project 
approach, budget support is often 
provided to countries with weak public 
financial management (PFM) systems. 
Given that the funds transferred 
under budget support operations are 
merged with other budget resources 
within the country’s budget (known as 
‘fungibility’), they are also exposed to 
the same PFM weaknesses, including 
risks of fraud and corruption76. In addi‑
tion, results achieved through budget 
support and the sustainability of such 
results can be put at risk in the event 
of political instability in the partner 
country, for example due to a lack of 
consideration for good governance, 
major economic and social issues or 
human rights. The decision to pro‑
vide EU budget support should thus 
be taken on a case‑by‑case basis and 
supported by an assessment of the 
expected benefits and potential risks77.

9
1
x
o
B

Examples of a lack of risk analysis for the chosen implementation modality

‘(…) the Commission does not convincingly demonstrate, before deciding to work with a UN organisation, 
that it has assessed whether the advantages offset any disadvantages. The choice of a UN organisation is not 
based on sufficient evidence that this approach is more efficient and effective than other ways of delivering 
aid.’78

78  Special Report No 15/2009, paragraph III.

 
37

79  Special Report No 4/2013.

80  Special Report No 9/2013.

81  Special Report No 14/2013.

82  Special Report No 13/2014 ‘EU 
support for rehabilitation 
following the earthquake in 
Haiti’ (http://eca.europa.eu).

4. Key risk areas

4.9.2. Exogenous factors

(b)  the development context in 

78 
Most EU development and coopera‑
tion actions are planned with a view 
to producing results over a period of 
several years. During this time, actions 
can experience significant changes or 
instability in:

(a)  the political or economic context 

at partner‑country level — this was 
the case, for example, in Egypt79, 
the Democratic Republic of the 
Congo80 and Palestine81 — or at 
project level (see Box 20);

respect of natural disasters, e.g. in 
Haiti, in January 2010, when the 
capital and the surrounding area 
were hit by an earthquake of mag‑
nitude 7.0 on the Richter scale82; or

(c)  EU aid management and control 

systems (see Box 21).

79 
Such changes in the context of EU 
development and cooperation actions 
can make it difficult to track and report 
on results in a consistent and mean‑
ingful way.

0
2
x
o
B

1
2
x
o
B

Example of instability in the economic context of a project

In Cambodia, the Commission financed a project to promote Waste to Energy technology in the rice milling 
sector. However, in recent years, Cambodia has seen major investments in larger and more efficient rice mills 
that have led to a four‑fold capacity increase between 2009 and 2013 83. It is now expected that most rice mill‑
ers require more powerful systems than those initially planned for promotion by the project, at much higher 
costs. For this reason, the project’s initial objectives have lost some of their relevance and are likely not to be 
achieved as initially planned.

83  World Bank policy note on Trade Development — Turning Cambodia into a leading rice exporter, September 2013.

Example of changes in management and control systems

On 1 January 2015, the Commission’s ‘Support Group for Ukraine’ and ‘Neighbourhood’ Directorate were 
transferred from EuropeAid to the Directorate‑General for Neighbourhood and Enlargement Negotiations, 
DG Neighbourhood and Enlargement Negotiations (formerly the Directorate‑General for Enlargement, DG 
Enlargement)84. This will affect both the geographical coverage of EuropeAid’s results framework and the way 
in which neighbourhood policy results are monitored and reported.

84  Minutes of the 2104th meeting of the Commission held on Wednesday 5.11.2014 — PV(2014) 2104.

 
 
5. Conclusions and 
recommendations

38

(f)  A lack of sufficient, relevant, 

reliable and up‑to‑date data can 
significantly weaken the quality 
and reliability of reported results 
(section 4.7).

(g)  Focusing on budgetary outturn as 
an objective can be detrimental to 
sound financial management and 
the achievement of results (section 
4.8).

(h)  Changes in the context of actions 

can undermine the achievement or 
sustainability of results, and make 
it difficult to assess the results 
achieved (section 4.9).

81 
Our review showed that these risk 
areas had been correctly identified 
by the Commission in a wide range of 
documents. It also showed the interest 
for the actions that are being under‑
taken by the Commission to launch 
an EU development and cooperation 
results framework.

82 
Nevertheless, further actions remain 
to be taken in order to harness the full 
potential of the Commission’s initia‑
tives for improved EU development 
and cooperation results.

5.1. Conclusions

80 
On the basis of our review, we iden‑
tified nine key risk areas related to 
a results‑oriented approach for EU 
development and cooperation action.

(a)  The use of inconsistent results‑re‑
lated terminology or failure to 
establish a clear logical chain 
between action, outputs, outcome 
and impact can be a source of con‑
fusion and undermine the assess‑
ment of what has been achieved 
(sections 4.1 and 4.2).

(b)  Increased complexity due to 

the integration of cross‑cutting 
issues in EU programmes repre‑
sents a supplementary risk for 
the achievement and reporting of 
results (section 4.3).

(c)  The lack of harmonisation between 
development partners’ aid delivery 
instruments, results frameworks 
and accountability structures can 
generate inefficiencies and ac‑
countability gaps (section 4.4).

(d)  Weaknesses in evaluation and 

results reporting can prevent the 
feedback of relevant knowledge 
and lessons‑learnt into reporting 
and decision‑making processes 
(section 4.5).

(e)  A lack of consolidated reports on 
or an overview of results achieved 
by EU aid can impair decision 
making and accountability 
(section 4.6).

5. Conclusions and recommendations

39

5.2. Recommendations

83 
In order to mitigate the risks identi‑
fied, we formulate the following 
recommendations.

Recommendation 1 
Improve the guidelines for 
terminology usage and 
formulation of objectives 
and indicators

The Commission should improve its 
guidelines to ensure the consistent 
use of terminology regarding results 
(outputs, outcomes, impacts) and, 
where relevant, to formulate SMART 
objectives and clear indicators for its 
interventions at all levels.

Recommendation 2 
Establish a clear link 
between actions and 
expected results

Recommendation 3 
Improve the information 
system for reporting 
results and lessons‑learnt

The Commission should improve its 
information system in order to be able 
to effectively consolidate and report 
on the results of its interventions. This 
should be complemented by measures 
enabling the Commission’s evaluation 
activities to identify useful lessons 
for subsequent legislation, program‑
ming or action design in the areas 
concerned.

Recommendation 4 
Ensure data availability 
and quality

In all its interventions, the Commis‑
sion should assess data availability and 
quality, and strive to have sufficient, 
relevant, reliable and up‑to‑date data 
on results achieved. This should be 
reflected in its systems and guidelines.

Through its internal guidelines, practi‑
cal tools, project formulation and 
monitoring processes, the Commission 
should ensure that a clear link is estab‑
lished between actions and expected 
results, including cross‑cutting issues.

Recommendation 5 
Assess the risks inherent 
to the implementation 
modality before 
committing financial 
resources

Before committing its financial re‑
sources, and when considering the 
expected results, the Commission 
should also assess the risks inherent to 
its choice of a particular implementa‑
tion modality.

5. Conclusions and recommendations

40

This report was adopted by Chamber III, headed by Mr Karel PINXTEN, Member of 
the Court of Auditors, in Luxembourg at its meeting of 17 November 2015.

For the Court of Auditors

Vítor Manuel da SILVA CALDEIRA
President

 
 
 
Annex

41

x Comprehensive Evaluation Framework for Budget Support Evaluations
e
n
n
A

c
i

e
l
b
a
n
i
a
t
s
u
S
.
t
c
a
p
m

I

.
5

h
t

w
o
r
g
e
v
i
s
u
l
c
n

i

d
n
a

n
o

i
t
c
u
d
e
r
y
t
r
e
v
o
p
d
n
a

m
o
n
o
c
e
e
v
i
s
u
l
c
n

i

d
n
a

e
l
b
a
n
i
a
t
s
u
s
d
e
c
n
a
h
n
 E
-

.

h
t

w
o
r
g

e
m
o
c
n

i

n

i

s
n
o

i
t
c
u
d
e
 R
-

-
n
o
n
d
n
a
y
t
r
e
v
o
p

.
y
t
r
e
v
o
p
e
m
o
c
n

i

d
n
a
t
n
e
m
r
e
w
o
p
m
 E
-

f
o
n
o
i
s
u
l
c
n

i

l
a
i
c
o
s

d
n
a
e
l
p
o
e
p
r
o
o
p

s
p
u
o
r
g
d
e
g
a
t
n
a
v
d
a
s
i
d

.
)
n
e
m
o
w
g
n
d
u
l
c
n
i
(

i

d
e
n
fi
e
d
s
a
s
e
u
s
s
i

r
e
h
t
 O
-

-
r
e
n
t
r
a
p
c
fi
i
c
e
p
s
e
h
t
n

d
n
a
s
k
r
o
w
e
m
a
r
f
p
h
s

i

i

-
e
v
o
r
p
m

i

.
g
.
e
(

s
e
i
t
i
r
o

i
r
p

-
n
o
r
i
v
n
e
,
s
t
h
g
i
r
n
a
m
u
h

,
y
c
a
r
c
o
m
e
d
n

i

s
t
n
e
m

.
)
n
o
i
t
c
e
t
o
r
p
t
n
e
m

I

)
Y
G
E
T
A
R
T
S
(
S
N
O
I
T
C
A
G
N
D
N
E
P
S
D
N
A
Y
C
I
L
O
P
T
N
E
M
N
R
E
V
O
G

e
c
i
v
r
e
s
–
s
e
i
r
a
i
c
fi
e
n
e
b

y
b
s
e
s
n
o
p
s
e
r
e
v
i
t
i
s
o
P

c
i

m
o
n
o
c
e
d
n
a
s
r
e
s
u

t
n
e
m
n
r
e
v
o
g
o
t
–
s
r
o
t
c
a

d
n
a
t
n
e
m
e
g
a
n
a
m
y
c
i
l

o
p

y
r
e
v
i
l
e
d
e
c
i
v
r
e
s

.
s
e
m
o
c
t
u
O

.
4

c
i
l

b
u
p
d
n
a
g
n
d
n
e
p
s

i

,
s
n
o

i
t
u
t
i
t
s
n

i

r
o
t
c
e
s

c
i
l

c
i
l

b
u
p

b
u
p

,
s
e
i
c
i
l

o
p
c
i
l

b
u
p
d
e
v
o
r
p
m

I

.
s
t
u
p
t
u
o
d
e
c
u
d
n

I

.
3

y
r
e
v
i
l
e
d
e
c
i
v
r
e
s

s
n
o

i

i
t
c
a
g
n
d
n
e
p
s
d
n
a
y
c
i
l

o
p
t
n
e
m
n
r
e
v
o
g
o
t

s
t
u
p
n

I

-
n
o

i
t
a
l
e
r
e
h
t
n

i

t
n
e
m
e
v
o
r
p
m

I

s
e
s
s
e
c
o
r
p
y
c
i
l

o
p
d
n
a
t
e
g
d
u
b

l
a
n
o

i
t
a
n
e
h
t
d
n
a
e
c
n
a
t
s
i
s
s
a

l
a
n
r
e
t
x
e
n
e
e
w

t
e
b
p
h
s

i

.
s
t
u
p
t
u
o
t
c
e
r
i
D

.
a
2

s
t
u
p
n

i
S
B
S
/
S
B
G

.
a
1

s
d
o
o
g
f
o
e
s
u
d
e
s
a
e
r
c
n

d
e
d
i
v
o
r
p
s
e
c
i
v
r
e
s
d
n
a

 I

r
o
t
c
e
s

c
i
l

b
u
p
e
h
t
y
b

g
n
i
t
l
u
s
e
r
d
e
c
n
a
h
n
e
d
n
a

e
t
a
v
i
r
p
d
n
a
e
c
n
e
d
fi
n
o
c

d
n
a
t
n
e
m

t
s
e
v
n

i

r
o
t
c
e
s

s
s
e
n
i
s
u
b
d
e
s
a
e
r
c
n

 I

n
o

i
t
c
u
d
o
r
p

-
e
v
i
t
i
t
e
p
m
o
c
d
e
v
o
r
p
m

 I

.
y
m
o
n
o
c
e
e
h
t

f
o
s
s
e
n

e
c
n
e
d
fi
n
o
c
d
e
v
o
r
p
m

 I

e
c
n
a
m
r
o
f
r
e
p
e
h
t
n

i

n
o

i
t
a
l
u
p
o
p
e
h
t

f
o

,
t
n
e
m
n
r
e
v
o
G
e
h
t

f
o

s
d
r
a
g
e
r

s
a
y
l
r
a
l
u
c
i
t
r
a
p

d
n
a
M
F
P

,
e
c
n
a
n
r
e
v
o
g

.
y
r
e
v
i
l
e
d
e
c
i
v
r
e
s

s
t
fi
e
n
e
b

-

-

-

-

c
i

m
o
n
o
c
e
o
r
c
a
m
d
e
v
o
r
p
m

 I

t
n
e
m
e
g
a
n
a
m

t
e
g
d
u
b
d
n
a

,
y
r
a
t
e
n
o
m

,
l
a
c
s
fi
s
a
h
c
u
s
(

d
n
a
y
t
i
t
n
a
u
q
d
e
s
a
e
r
c
n

 I

d
n
a
s
d
o
o
g
f
o
y
t
i
l
a
u
q

e
h
t
y
b
d
e
d
i
v
o
r
p
s
e
c
i
v
r
e
s

c
i

m
o
n
o
c
e
d
n
a
e
d
a
r
t

.
)
s
e
i
c
i
l

o
p
h
t

w
o
r
g

-

-

r
o
t
c
e
s

c
i
l

b
u
p

n
o

i
t
u
c
e
x
e
d
n
a
n
o

-
i
c
s
i
d

l
a
c
s
fi

s

m
e
t
s
y
s

,
y
c
n
e
r
a
p
s
n
a
r
t
(

t
n
e
m
e
r
u
c
o
r
p

e
v
i
t
a
c
o

l
l
a
,
t
h
g
i
s
r
e
v
o
,
e
n

i
l

)
y
c
n
e
i
c

ffi
e
l
a
n
o

i
t
a
r
e
p
o
d
n
a

p

y
c
i
l

o
p
c
i
l

b
u
p
d
e
v
o
r
p
m

 I

-

d
n
a
M
F
P
d
e
n
e
h
t
g
n
e
r
t
 S
-

i
t
a
l
u
m
r
o
f

s
e
s
s
e
c
o
r
p

r
o
t
c
e
s

c
i
l

b
u
p
d
e
n
e
h
t
g
n
e
r
t
 S
-

s
k
n

i
l

d
e
n
e
h
t
g
n
e
r
t
 S
-

.
s
n
o

i
t
u
t
i
t
s
n

i

t
n
e
m
n
r
e
v
o
G
e
h
t
n
e
e
w

t
e
b

n

i

s
e
i
d
o
b
t
h
g
i
s
r
e
v
o
d
n
a

n
o

i
t
a
l
u
m
r
o
f
y
c
i
l

o
p
f
o
s

m
r
e
t

d
n
a
l
a
i
c
n
a
n
fi

,
l
a
v
o
r
p
p
a
d
n
a

-
l
i

b
a
t
n
u
o
c
c
a
l
a
i
c
n
a
n
fi
-
n
o
n

,

n
o

i
t
a
s
i
l
a
r
t
n
e
c
e
d
d
e
c
n
a
h
n
e

y
n
i
t
u
r
c
s

t
e
g
d
u
b
d
n
a
y
t
i

n

i

s
t
n
e
m
e
v
o
r
p
m

i

r
e
h
t
 O
-

.

g
.
e
(

s
e
u
s
s
i
e
c
n
a
n
r
e
v
o
g

,

w
a
l

f
o
e
l
u
r

f
o
n
o

i
t
a
c
i
l

p
p
a

)
s
t
h
g
i
r
n
a
m
u
h

f
o
e
r
a
h
s
d
n
a
e
z
i
s
d
e
s
a
e
r
c
n

 I

s
d
n
u
f
e
c
n
a
t
s
i
s
s
a
l
a
n
r
e
t
x
e

h
g
u
o
r
h
t
e
l
b
a
l
i
a
v
a
e
d
a
m

e
r
a
h
s
d
n
a
e
z
i
s
d
e
s
a
e
r
c
n

r
o
f
e
l
b
a
l
i
a
v
a
t
e
g
d
u
b
f
o

 I

.

i

g
n
d
n
e
p
s
y
r
a
n
o

i
t
e
r
c
s
i
d

.
t
e
g
d
u
b

l
a
n
o

i
t
a
n
e
h
t

f
o
y
t
i
l
i

b
a
t
c
i
d
e
r
p
d
e
s
a
e
r
c
n

 I

-
r
e
t
x
e
f
o
t
n
e
m
e
s
r
u
b
s
i
d
e
h
t

-

-

-

.
s
d
n
u
f

l
a
n

y
t
i
c
a
p
a
c
/
A
T
d
n
a
s
e
i
t
i
l
a
n
o

i
t

r
e
t
t
e
b
s
e
i
t
i
v
i
t
c
a
g
n
d

i

l
i

u
b

-
n
o
c
e
r
o
m
d
n
a
d
e
t
a
n
d
r
o
o
c

i

n
o

i
t
a
t
n
e
m
e
l
p
m

i

r
o
f
e
v
i
c
u
d

.
s
e
i
g
e
t
a
r
t
s

t
n
e
m
n
r
e
v
o
g
f
o

-
t
e
b
)
S
B
g
n
d
u
l
c
n
i
(
e
l

i

o
h
w
a

d
e
n
g

i
l
a
d
n
a
d
e
s
i
n
o
m
r
a
h
r
e
t

d
n
a
s
e
i
c
i
l

o
p
t
n
e
m
n
r
e
v
o
g
o
t

s
a
e
c
n
a
t
s
i
s
s
a
l
a
n
r
e
t
x
 E
-

-
i
d
n
o
c

,
e
u
g
o

l
a
i
d
y
c
i
l

o
 P
-

.
s

m
e
t
s
y
s

f
o
s
t
s
o
c
n
o

i
t
c
a
s
n
a
r
t
d
e
c
u
d
e
 R
-

.

i

d
i
a
g
n
d
i
v
o
r
p

d
e
s
a
b
y
r
u
s
a
e
r
T
l
a
n
o

i
t
a
n

d
e
e
r
g
a
y
l
s
u
o

i
v
e
r
p
n
o

e
h
t
o
t

s
d
n
u
f

f
o
r
e
f
s
n
a
r
 T
-

s
e
i
t
i
l
a
n
o

i
t
i
d
n
o
c

s
r
o
t
a
c
i
d
n

i
e
c
n
a
m
r
o
f
r
e
p

-
i
v
i
t
c
a
g
n
d

i

l
i

u
b
y
t
i
c
a
p
a
 C
-

l
a
c
i
n
h
c
e
t
g
n
d
u
l
c
n

i

i

s
e
i
t

d
n
a
e
u
g
o

l
a
i
d
y
c
i
l

o
 P
-

.
e
c
n
a
t
s
i
s
s
a

s
u
o

i
r
a
v
y
b
s
t
c
e
ff
e
r
e
h
t
O

.

b
2

s
t
u
p
n

i

t
n
e
m
n
r
e
v
o
G

t
n
e
m
n
r
e
v
o
G
s
u
o

i
r
a
V

.

b
1

s
t
u
p
n

i

s
t
u
p
n

i
y
c
i
l

o
p
c
i
t
s
e
m
o
d
d
n
a

i

g
n
d
n
u
f
e
u
n
e
v
e
r

c
i
t
s
e
m
o
 D
-

x
x
x
x
-

r
e
h
t
o
y
b
s
t
c
e
ff
e
r
e
h
t
O

.
c
2

e
c
n
a
t
s
i
s
s
a
l
a
n
r
e
t
x
e

r
e
h
t
o
f
o
s
t
u
p
n

I

.
c
1

e
c
n
a
t
s
i
s
s
a
l
a
n
r
e
t
x
e

s
e
m
m
a
r
g
o
r
p

EXTERNAL FACTORS, CONTEXT FEATURES AND FEED BACK PROCESSES

© OECD — DAC Network on Development Evaluation — Evaluating Budget Support Methodological Approach.

s
l

o
o
t

s
e
v
i
t
n
e
c
n

i

i

g
n
g
n
a
h
c
o
t

s
e
s
n
o
p
s
e
R
-

y
t
i
l
i

b
a
t
n
u
o
c
c
a
c
i
t
s
e
m
o
d
f
o
h
t
g
n
e
r
t
S
-

s
e
s
s
e
c
o
r
p
m
r
o
f
e
r
o
t

t
n
e
m

t
i

m
m
o
c

l
a
c
i
t
i
l

o
p
f
o
t
n
e
t
x
 E
-

s
l

o
o
t
d
n
a
s
e
s
s
e
c
o
r
p
g
n
n
r
a
e
l

i

g
n
i
t
s
i
x
E
-

t
n
e
m
p
o

l
e
v
e
d
c
i

m
o
n
o
c
e
l
a
b
o
G
-

l

r
o
t
c
e
s

c
i
l

b
u
p
f
o
y
t
i
c
a
p
a
C
-

w
o
fl
n

i

l
a
t
i
p
a
c
n
g
i
e
r
o
F
-

s
e
c
i
v
r
e
s

t
v
o
G
r
o
f
d
n
a
m
e
d
f
o
e
r
u
t
a
N
-

t
n
e
m
e
l
p
m

i

o
t
y
t
i
c
a
p
a
c

t
n
e
m
n
r
e
v
o
G
-

’
s
n
o

i
t
i
d
n
o
c
y
r
t
n
e

‘

e
h
t

f
o
s
e
r
u
t
a
e
f

s
u
o

i
r
a
V
-

;
s

m
r
o
f
e
r

k
r
o
w
e
m
a
r
f
d
i
a
l
l
a
r
e
v
O
-

x
x
x
x

-

x
x
x
x
-

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Reply of the  
Commission

42

Executive summary

Framework as one of the tools for monitoring and 
reporting on the implementation of the MFF. 

III
The nature of the Court’s review is such that it 
includes examples which sometimes relate to 
actions undertaken several years ago. This is in 
particular true in relation to the examples relating 
to budget support, for which a number of risks have 
been addressed through the adoption of revised 
budget support guidelines in 2012 (see the replies 
provided to various of the boxes hereafter), as well 
as for the examples relating to multiannual pro‑
gramming documents (strategy papers or indicative 
programmes), for which the Commission would 
like to point out that it has systematically included 
indicators in the programming documents for the 
period 2014‑2020.

1. Introduction

07
The Results Framework (RF) is one of the tools for 
reporting results at output and direct outcome lev‑
els. Evaluations provide judgements on the causal, 
logical chain between the different level of results 
up to the impact and the achievement of ‘strate‑
gic development objectives’ be it at country or at 
global level. Furthermore, several initiatives have 
been taken to improve the whole Monitoring and 
Evaluation (M&E) system to improve the Commis‑
sion's results‑oriented approach.

10
The Staff Working Document (SWD) on the ‘MORE 
framework’ describes the various existing frame‑
works for monitoring, evaluation and reporting on 
the implementation of the multiannual financial 
framework (MFF). In terms of objectives and indica‑
tors, it describes what has been set up in the legal 
acts establishing 2014‑2020 financing instruments 
and complemented under the 2014 and 2015 budg‑
etary programme statements as well as the moni‑
toring, evaluation and reporting tools as included 
in the Annual Management Plans (AMP) and 
Annual Activity Reports (AAR). For external assis‑
tance instruments, it also refers to the new Results 

3. Our review

22
The nature of the Court’s review is such that it 
includes examples which sometimes relate to 
actions undertaken several years ago. This is in 
particular true in relation to the examples relating 
to multiannual programming documents (strategy 
papers or indicative programmes), for which the 
Commission would like to point out that it has sys‑
tematically included indicators in the programming 
documents for the period 2014‑2020, as well as in 
relation to the examples relating to budget support, 
for which a number of risks have been addressed 
through the adoption of revised budget support 
guidelines in 2012. 

The Commission also refers to its replies to the 
Court’s special reports in question and to the addi‑
tional replies to the Court’s observations in the pre‑
sent report with respect to some of the examples 
so as to underline one or more aspects it considers 
more specifically relevant in the context of its reply 
to those observations.

4. Key risk areas

26
The Commission is aware and agrees with the Court 
on the need to ensure consistency in terms of the 
terminology used. In the Staff Working Document 
(SWD) of 26 March 2015 ‘Launching the EU Inter‑
national Cooperation and Development Results 
Framework’, the terminology input‑output‑out‑
come‑impact is used, corresponding to the inter‑
nationally practised OECD‑DAC results terminology 
and being also in line with the one used in the 
2014‑2020 financing instruments. The same termi‑
nology is used in the new Action Document tem‑
plate (applicable as of 1st January 2015) and in the 
new Results Oriented Monitoring (ROM) Handbook 
(‘Handbook on ROM reviews and support to end‑of‑
project results reporting’) (March 2015). 

Reply of the Commission 

43

Box 1
The terminology used in EuropeAid’s Annual Man‑
agement Plan is following the one foreseen in the 
Commission’s Secretariat‑General and DG Budget 
overall instructions for the establishment of the 
management plans by the Commission’s general 
directorates and services. 

The risk as set out in the example given in Box 
1 refers to the terminology used in the Results 
Oriented Monitoring (ROM) Handbook applicable 
to the previous ROM system which ended in spring 
2014. It has been replaced by a new ROM system 
and a new ROM Handbook introduced in spring 
2015, which no longer contains such reference and 
is fully in line with internationally practiced lan‑
guage applying the reference to SMART objectives 
and not to indicators. Accordingly, the Commission 
will modify the next budget circular.

Moreover, the Commission intends to refine the 
language and instructions related to the objec‑
tives and indicators to be used in the context of the 
budgetary circular.

The Commission agrees that objectives and 
expected results should as much as possible be 
SMART but also considers that this is in many cases 
not easy to implement. This is particularly true 
where expected results can only reasonably be 
measured in qualitative terms and cannot be quan‑
tified which is inter alia the case for results of broad 
policies or financing instruments.

Corporate results frameworks, by nature, focus 
on quantitative results, as mentioned in the Staff 
Working Document ‘Launching the EU International 
Cooperation and Development Results Framework’ 
(EU RF). Indicators in the EU RF have been set as 
much as possible following RACER criteria (‘Rel‑
evant, Accepted, Credible, Easy, and Robust’) — as 
set out in Eurostat’s indicator guidance, referred to 
by the Court.

29
The Commission agrees that the broad variety of 
the documents referred to by the Court increases 
the risk of inconsistencies. Taking into account that 
the documents are adopted at different stages of 
policy formulation and the project and programme 
management cycle (including the programming 
phase) it necessarily results in adjustments and 
more specific types of focus in implementation. 

The Commission has taken measures to reduce the 
risks mentioned by the Court. Thus, in the 2014‑
2020 programming exercise, country, regional and 
thematic programming documents have been 
systematically reviewed in order to ensure expected 
results and indicators are of appropriate quality (i.e. 
measurable) and of a manageable number (see the 
programming documents at http://eeas.europa.eu/
around/index_en.htm). 

The revision of the Action Document template (in 
force as of 1 January 2015) and the ongoing revision 
of the related internal Quality Support Group docu‑
ments aim to ensure that expected results and indi‑
cators are properly included in Action documents. 

Furthermore, the Commission’s services are working 
on the improvement of the Monitoring and Evalu‑
ation system, including room for adjustment that 
may be needed over implementation of the actions 
and the flexibility that is required to implement 
such adjustments in order to adapt to changing 
country contexts and policies. 

32
The Commission has taken and is taking measures 
to reduce the risks mentioned by the Court. See 
Commission reply to paragraph 29.

Box 4
In compliance with recommendation 2 on the 
development of performance indicators made by 
the Court in the Special Report 14/2013, the Com‑
mission developed a results‑oriented framework 
that includes indicators to monitor the progresses 
of the Palestinian Authority in its reforms.

Reply of the Commission 

44

The results‑oriented framework, which has been 
developed in coordination with the World Bank and 
with EU Member States in the framework of the EU 
Local Development Strategy, was agreed with the 
Palestinian Authority through a Memorandum of 
Understanding signed on 25 March 2015. The areas 
covered in the pilot phase include Macroeconomic/
Fiscal indicators, PFM, public administration reform 
(including measures addressing the reintegration 
of non‑working Gaza civil servants), education, 
health and social protection. An evaluation of the 
pilot phase will take place end 2015/early 2016 and 
additional sectors could be added for the next 
phase. The text above shows that indicators have 
been established, which is in line with the ENPI 
regulation.

Box 5
The Commission sees the case as an example where 
it ultimately had to adapt the understanding of the 
indicator to the conditions of specific local context 
and accordingly to accept the adaptation of the 
target value of the indicator. The example demon‑
strates the importance of being continuously atten‑
tive to such situations but does not relate to a risk 
which the Commission can manage otherwise than 
in that way.

38
Institutional reforms are a long and difficult process 
where helping the government of a partner country 
to progress step by step can be more sustainable 
than a quick win of project. See reply to Box 6.

Box 6
The Commission notes that the quoted payments 
(referring to June 2008, December 2010, July 2011) 
largely precede the new policy on budget support 
which introduced a much more rigorous assess‑
ment of eligibility criteria, notably the PFM criterion. 
The risk which is to be illustrated by this example is 
reduced through and since the application of the 
2012 budget support guidelines.

In its reply to the Court's findings the Commission 
also noted that ‘A distinction must be made, on the 
one hand, between the eligibility criteria for budget 
support, which require a holistic approach based on 
credibility and positive implementation and which 
are by nature general, and, on the other hand, the 
specific conditions, attached to variable tranches, 
which must indeed be clear and specific …. Donors 
must acknowledge the complexity of reforms and 
define targets that are more reasonable and better 
sequenced. For example, progress in PFM reform 
in both countries, Kyrgyzstan and Egypt, receiving 
budget support has been slower than expected. 
Indeed PFM reform plans, initially largely prepared 
by external donors, were far too ambitious and 
underestimated the local capacities to implement 
them. To remedy this, PFM reforms and action plans 
were revised to be more realistic and additional 
Technical Assistance was provided.’

As a result, the case is rather an example of overly 
ambitious objectives. In addition, it has to be 
underlined that the reform of the system of public 
finance management is a long‑term process. Within 
the timespan of one operation of 3 years, only very 
limited results can be detected.

The case of Egypt rather relates to an example 
where the issue is not about defining a partner 
country’s commitment to implement reforms as 
an objective (see title of section 4.1.4) but, in terms 
of the 2012 budget support guidelines, about the 
precondition for disbursement related to essential 
elements like human rights.

Following the Council conclusions of August 2013/ 
February 2014 and the recommendations of the 
Court, the Commission has focused its assistance on 
socioeconomic interventions and support to civil 
society.

There has been no new budget support operation 
in Egypt since 2011, and no budget support dis‑
bursement has been made on any of the five ongo‑
ing budget support operations since 2012.

Reply of the Commission 

45

Box 7 — Second indent 
The budget support operations in Malawi and 
Rwanda are dating before 2010 and largely precede 
the new policy on budget support which intro‑
duced a much more rigorous approach to clarifying 
the link between the operation financed by the EU 
and the national policies.

The risk which is to be illustrated by this example is 
reduced through and since the application of the 
2012 guidelines.

Box 8 
The operations targeted emergency budget sup‑
port measures to allow macro‑economic stabilisa‑
tion of the country with the global objective of 
sustaining the Democratic Republic of Congo in the 
economic and food crisis, and consequently, the 
results analysis has to be seen in this context.

The PFM reform plan the Court refers to was indeed 
mentioned in the Financing Agreement and its 
establishment was one of the milestones to meas‑
ure progress.

Fixed tranches are also subject to eligibility, i.e. 
performance related. Therefore the payment 
conditions were linked to objectives including PFM 
improvements.

46
EU contributions to country development results 
will be reported as country results supported, thus 
opting for a contribution approach (see Com‑
mission’s March 2015 Staff Working Document 
‘Launching the EU International Cooperation and 
Development Results Framework’). To quantify such 
contribution, two methods which are currently 
being used by donors are possible: 

—  the ‘overall method’, where the total (or overall) 
results the EU achieved jointly with others (such 
as the partner country government and other 
donors) would be reported, rather than results 
linked to the share of funding that has been 
provided by the EU; 

—  the ‘proportional method, where results would 
be reported based on the EU’s proportion of 
total funding. Under this approach results of 
projects and programmes are calculated as 
a pro rata share equal to the level of financial 
inputs provided by the EU. In this approach, the 
reporting proceeds implicitly on a direct causal 
link between funding provided and results 
achieved.

At an aggregate level, like in the case of report‑
ing on the basis of the new results framework, one 
method needs to be chosen. As detailed budget 
information on inputs by others, including part‑
ner governments, is not always easily available to 
allow proportional results to be calculated, the 
Commission has opted for reporting following the 
overall approach as being for the time being the 
only realistic option to report results achieved in 
an aggregated way. As equally set out in the March 
2015 Staff Working Document, the overall method 
would also be more appropriate from an aid effec‑
tiveness perspective, with the strong focus it places 
on country ownership.

Reply of the Commission 

46

Box 9 
Whereas it is possible to assess in general whether 
a national policy to which a budget support pro‑
gramme provided support, has contributed to cer‑
tain well defined results achieved at country level, 
it is not possible to attribute a particular proportion 
of the result to the EU funding.

Box 11 
The example set out in Box 11 rather focuses on 
the insufficiency of integrating the human rights 
perspective in programmes than on the increased 
complexity due to the integration of cross‑cutting 
issues or on the additional resulting risks for appro‑
priate definition of objectives and indicators and 
related results measurement.

Box 14 — Second indent 
In 2015, the Commission has introduced specific 
provisions on results reporting by EU implementing 
partners in the general conditions of the Admin‑
istration Agreement (covering the World Bank as 
implementing partner) and of the ‘PAGODA’ Agree‑
ment covering all pillar assessed organisations — 
including UN Agencies. This should ensure greater 
availability of results data in progress and final 
reports transmitted by EU implementing partners.

59
The fact that in certain cases, by the nature of the 
project or programme, the results can only be 
identified and reported at the end of the project or 
programme, and therefore do not allow for periodic 
assessment of their achievement, does not inevita‑
bly lead to weak reporting on results. In such cases 
milestones or benchmarks can be identified which 
assist in tracking whether the project or programme 
is well under way to the effective achievement of 
results.

60
The Commission has already taken steps to improve 
evaluations by:

—  drafting and issuing an ‘Evaluation Policy for 

the European Union development cooperation’ 
(2014);

— 

— 

improving the quality of the planning, imple‑
mentation, monitoring of evaluations of pro‑
jects and programmes (ongoing); 

increasing the involvement of the management 
in strategic evaluations already at the Inception 
phase (ongoing); and

—  applying the ‘Evaluation first principle’ in ac‑

cordance with the Communication COM(2013) 
686 final.

Box 15 
The Commission does not consider the example as 
a case of weakness in results reporting or evaluation 
with respect to the action as a whole, the effective‑
ness of which can per definition only be assessed 
towards the end of or after the implementation of 
all its components or sub‑projects.

65
In 2015, the Commission will issue its first consoli‑
dated results report in relation to the assistance 
provided within the context of the EU international 
cooperation and development policies. This will 
take the form of a complementary report to the 
Annual report. 

Box 16 — Third indent 
Improved monitoring and reporting systems and 
the implementation of an operational informa‑
tion management system (replacing also CRIS) will 
significantly step up EuropeAid’s use of informa‑
tion systems for reporting, in particular results 
reporting.

Reply of the Commission 

47

Box 17
The weaknesses of national data collection and 
information management systems in the countries 
to which the EU provides assistance is a factor 
which is inherent to the developing status of the 
countries concerned and a permanent point of con‑
cern on all donors’ side. This can only be sustainably 
addressed by building up appropriate national sta‑
tistical systems. This risk factor is mitigated on the 
EU side as part of the sector and other support the 
EU is providing. It can however only be addressed 
sufficiently by sustained collective efforts of the 
country and the donor community. 

72
The MFF 2014‑2020 introduces a number of new 
features particularly to stimulate spending pro‑
grammes' performance. Building on this, a Com‑
mission initiative lead by Vice‑President Kristalina 
Georgieva, Budget and Human Resources aims at 
a shift from spending to an EU Budget Focused on 
Results. The objective is to create the conditions for 
further strengthening the budget’s performance 
in terms of both programme results and sound 
financial management of funds in a balanced, com‑
prehensive approach. This aims at reducing the risk 
of focusing on budgetary implementation to the 
detriment of programme results, including spend‑
ing areas and methods by which the EU budget can 
maximise results and rigorous application of cost‑
effective controls for enhanced compliance.

73
The Commission addresses the risk by multian‑
nual programming, by 24 months planning of the 
implementing individual actions and by the internal 
quality support group process. These actions are 
complemented by further measures such as the 
upcoming EU Results Framework. See also reply to 
paragraph 29.

75
For budget support programmes, the Commission 
has developed a risk management framework (RMF) 
to identify major risks and mitigate them. The RMF 
is taken into account in the decision‑making pro‑
cess notably in the enhanced governance through 
the internal Budget Support Steering Committee. 
The RMF informs systematically the policy dialogue 
with partner countries on the strategic level, as well 
as by focussing on key issues. The RMF is used to 
balance expected benefits, as defined in the action 
document, with identified risks and proposes miti‑
gating measures.

Internal analysis (budget support report 
2013 and 2014) shows that EU budget support 
countries clearly show better and improving PFM 
performance compared with non‑budget support 
aid recipients.

The risk which is to be illustrated by this example is 
reduced through and since the application of the 
2012 budget support guidelines.

Box 19
In 2015, the Commission has decided to step up 
on a case‑by‑case basis the underlying assess‑
ment whether or not to work with an international 
organisation as an implementing partner.

Box 20 
The Commission recognises that changes in the 
political and economic context represent risks in 
terms of results to be attained. This type of risks are 
the reason why the context analysis is an important 
feature of the standard identification and formula‑
tion process applied by the Commission’s services 
and EU Delegations in the preparation of projects 
and programmes.

Reply of the Commission 

48

In order to adapt to unforeseen changes of the aid 
context a certain degree of flexibility is needed 
during the implementation phase. This has been 
ensured by simplifying relating rules and proce‑
dures as of July 2014. These simplification measures 
included changes relating to the use of indicators 
so that modifications of specific priorities or in rela‑
tion to the collection and availability of data can be 
taken into account.

It can however not be excluded that the project’s 
initial objectives cannot be achieved, in which case 
they need to be adapted to the changing circum‑
stances and context.

Box 21 
The reorganisation of January 2015 will as such 
affect neither the definition of objectives nor the 
attainment of related results within the framework 
of specific actions. 

The Commission, irrespective of its own administra‑
tive organisation, and in line with the provisions of 
the common implementing regulation covering the 
ENI countries (in particular articles 12 and 13), will 
respect its obligations to monitor the implementa‑
tion of projects and report to the Member States, 
the EP and other EU institutions.

The Commission will be reporting on the basis of 
the results framework also for the Neighbourhood 
countries, consistent with the Council conclusions 
n. 9145/15 of 26.05.2015. A specific tool to provide 
support to ‘end of project results reporting’, has 
been put in place within the framework of the new 
Result Oriented Monitoring (ROM) system to help 
consolidating data on end of project results data, in 
particular in relation to the indicators based on the 
agreed results framework. 

As regards evaluation, DG NEAR has its own Evalu‑
ation Function, addressing since 2015 both the 
Neighbourhood and the Enlargement regions.

5. Conclusions and 
recommendations

80(a)
At various levels, the Commission services have 
been working on ensuring consistent terminology: 
the sector indicator guidance 2013, the action docu‑
ment template and related guidance applicable as 
of 1 January 2015, as well as the new International 
cooperation and development results framework 
and the related instructions and guidance for 
reporting on the basis of the results framework.

The Commission services are furthermore in the 
process of strengthening support to internal quality 
assurance with respect to these aspects of individ‑
ual action documents, considered important by the 
Commission both for monitoring and reporting on 
implementation, performance and results and for 
the evaluation of the actions.

They are all also in line with the terminology used in 
the basic regulations which constitute the regula‑
tory framework for the EU external assistance.

The Commission agrees that establishing ‘a clear 
logical chain’ from inputs to impact is of utmost 
importance at project level. This has been rein‑
forced with the introduction of the Action Docu‑
ment template as of 1 January 2015. However, 
establishing a clear logical chain at strategic or at 
policy level will never reflect the complexity of 
development context. In the international evalua‑
tion world, the use of the linear logical framework 
at such strategic/policy level is therefore more and 
more challenged. 

80(b)
The integration of cross‑cutting issues presents 
indeed several challenges to appropriate formula‑
tion of the specific objectives and results of the 
actions and to the collection of corresponding data 
allowing to monitor their implementation.

Reply of the Commission 

49

80(c)
The risk of inefficiencies and gaps to which the 
Court points would seem most present at coun‑
try level. This demonstrates the importance of 
the alignment by donors on the country system/
policy/results framework as the preferred approach 
compared to only harmonising among donors, as 
already pointed out by the Court in paragraph 17(a) 
and (b). 

By applying this approach, the Commission has 
already been making important efforts in the con‑
text of the 2014‑2020 multiannual programming to 
align as much as possible on country policies and 
frameworks, in accordance with the Busan com‑
mitments on aid effectiveness. This has also been 
the Commission’s approach in setting up the new 
corporate results framework. At country level, the 
Commission is together with other donors in favour 
of the definition of country‑owned results frame‑
works and sees joint programming with EU Mem‑
ber States as one of the means to promote such 
frameworks.

80(h)
Changes in the context of actions may lead to 
changes of the indicators used or of the initial tar‑
get values of the indicators.

Recommendation 1
The Commission accepts the recommendation and 
has already been working on ensuring consistent 
terminology: the sector indicator guidance 2013, 
the action document template and related guid‑
ance applicable as of 1 January 2015, as well as the 
new International cooperation and development 
results framework and the related instructions and 
guidance for reporting on the basis of the results 
framework.

They are all in line with the terminology used in the 
basic regulations.

The Commission services are furthermore in the 
process of strengthening support as part of internal 
quality assurance with respect to these aspects of 
individual action documents, and will also continue 
to improve its general guidance on project and pro‑
gramme management and staff training modules in 
this area.

Recommendation 2
The Commission accepts the recommendation.

It has been in the process over the past 2 years of 
preparing various measures to strengthen its sys‑
tems, guidance and processes in this respect. Please 
see also reply to Recommendation 1.

The Commission has agreed (see reply to paragraph 
80 a) that establishing ‘a clear logical chain’ from 
inputs to impact is of utmost importance at project 
level.

Recommendation 3
The Commission accepts the recommendation.

Actions are under way to improve the whole 
monitoring and evaluation system, enhancing also 
the Commission’s capacity to identify lessons (see 
also the Commission’s replies to the Court’s Special 
Report 18/2014). 

As regards more specifically the information sys‑
tem, the actions the Commission has been taking 
so far is the establishment of a preliminary system 
to ensure consolidated results reporting for its first 
results report in 2015 prior to the development of 
the first modules of its new operational informa‑
tion system which are planned to comprise also the 
reporting on results.

Reply of the Commission 

50

Anticipating the new system and in line also with one of 
the recommendations of the Court in its Special Report 
18/2014 on Evaluations and ROM, the Commission is in 
the final stage of the development of a specific informa‑
tion module relating to evaluations, foreseen to be put 
into production in the first half of 2016, which will pro‑
gressively improve the Commission’s capacity to identify 
lessons learnt for the various relevant purposes.

Recommendation 4
The Commission accepts the recommendation. It has 
already stepped up its efforts in that sense as mentioned 
in its reply to Recommendation 1. 

Recommendation 5
The Commission accepts the recommendation. 

The implementation of the recommendation will be part 
of the general assessment of the risks inherent to an 
action (e.g. choice of a particular implementation modal‑
ity, please see also section 2 ‘Risks and assumptions’ of 
the template of the Action Document applicable since 
1 January 2015).

For budget support programmes, the Commission has 
developed a risk management framework (RMF) to 
identify major risks and mitigate them. The RMF is taken 
into account in the decision‑making process notably in 
the enhanced governance through the internal Budget 
Support Steering Committee. The RMF informs systemati‑
cally the policy dialogue with partner countries on the 
strategic level. The RMF is used to assess the balance to 
be ensured between expected benefits of the envisaged 
action, to identify risks and to propose risk‑mitigating 
measures.

Internal analysis (budget support report 2013 and 2014) 
shows that EU budget support countries clearly show 
better and improving PFM performance compared with 
non‑budget support aid recipients.

In 2015, the Commission has decided to step up the 
underlying assessment whether or not to work with an 
international organisation as an implementing partner 
on a case‑by‑case basis.

HOW TO OBTAIN EU PUBLICATIONS

Free publications:

(cid:127) 

one copy: 
via EU Bookshop (http://bookshop.europa.eu);

(cid:127)   more than one copy or posters/maps: 

from the European Union’s representations (http://ec.europa.eu/represent_en.htm);  
from the delegations in non-EU countries (http://eeas.europa.eu/delegations/index_en.htm);  
by contacting the Europe Direct service (http://europa.eu/europedirect/index_en.htm) or  
calling 00 800 6 7 8 9 10 11 (freephone number from anywhere in the EU) (*).

(*) 

The information given is free, as are most calls (though some operators, phone boxes or hotels may charge you).

Priced publications:

(cid:127) 

via EU Bookshop (http://bookshop.europa.eu).

 
In our review, we identified nine key risk areas related to 
a results-oriented approach for EU development and 
cooperation action.
Our review showed that these risk areas had been 
correctly identified by the Commission. It also showed the 
interest in the Commission’s initiatives to improve EU 
development and cooperation results.
Nevertheless, further actions remain to be taken in order 
to make these initiatives fully effective. We formulate 
a number of recommendations in this regard.

EUROPEAN
COURT
OF AUDITORS