Special Report No 18 // 2014#EuropeAid’s evaluation and resultsoriented monitoring systems #(pursuant to Article 287(4), second subparagraph, TFEU)
Date: 2014-12-12
Institutions: European Court of Auditors
Subjects: EU aid, EU expenditure, EU office or agency, aid evaluation, development aid, evaluation method, humanitarian aid, project evaluation

EN 

2014 

NO 18

Special Report  EuropeAid’s evaluation 

and results‑oriented 
monitoring systems

EUROPEAN
COURT
OF AUDITORS

 
EUROPEAN COURT OF AUDITORS 
12, rue Alcide De Gasperi 
1615 Luxembourg 
LUXEMBOURG

Tel. +352 4398-1

E-mail: eca-info@eca.europa.eu 
Internet: http://eca.europa.eu

Twitter: @EUAuditorsECA 
YouTube: EUAuditorsECA

More information on the European Union is available on the Internet (http://europa.eu).

Luxembourg: Publications Office of the European Union, 2014

ISBN 978-92-872-1898-8
doi:10.2865/44910

© European Union, 2014
Reproduction is authorised provided the source is acknowledged.

Printed in Luxembourg

EN 

2014 

NO 18

Special Report

EuropeAid’s evaluation 
and results‑oriented 
monitoring systems

(pursuant to Article 287(4), second subparagraph, TFEU)

Contents

02

Paragraph

Glossary and abbreviations

I–VII 

Executive summary

1–10 

Introduction

2–5 

The Commission’s evaluation framework

6–10 

EuropeAid’s evaluation and ROM systems

11–13  Audit scope and approach

14–64  Observations

14–22  Overall, the evaluation and ROM functions are well organised but EuropeAid could pay more 

attention to the efficient use of resources

15–17 

18–22 

There is a sound distribution of responsibilities but insufficient supervision of programme evaluations

EuropeAid does not give enough consideration to the efficient use of resources

23–52 

EuropeAid’s evaluation and ROM systems do not sufficiently ensure relevant and robust findings

24–31 

32–38 

39–52 

There are some weaknesses in the selection of programme evaluations and timely delivery of evaluations

The procedures to ensure the quality of programme evaluations and ROMs are implemented inconsistently

The evaluation and ROM systems do not provide adequate information on results achieved

53–64 

EuropeAid’s evaluation and ROM systems do not ensure the maximum use of the findings made

54–58 

59–64 

Proper follow‑up mechanisms are not in place

Evaluation and ROM findings are well disseminated except for programme evaluations

65–70  Conclusions and recommendations

 
Contents

03

Annex I  —  ROM grading system

Annex II  —  List of programming documents reviewed

Annex III  —  List of financing agreements reviewed

Annex IV  —  List of evaluations reviewed

Annex V  —  List of ROMs reviewed

Annex VI  —  List of Delegations which replied to the survey

Reply of the Commission

 
 
 
 
 
 
 
Glossary  
and abbreviations

04

Annual Report on the European Union’s Development and External Assistance Policies and their 
Implementation (AR): This report is requested in the regulations establishing the financing instruments for the 
EU’s external assistance and has to be submitted to the European Parliament, the Council, the European Economic 
and Social Committee and to the Committee of the Regions. It is a report on the achievement of the objectives of 
each financing instrument, is based on indicators and measurements of the results delivered and the efficiency of 
the relevant instruments. It can be found at: http://ec.europa.eu/europeaid/annual‑report‑2014‑european‑unions‑ 
development‑and‑external‑assistance‑policies‑and‑their_en

Annual Activity Report (AAR): This report presents the achievements reached and resources used by the 
Directorate‑General. It is the ‘mirror’ of the annual work programme. It is also a management report of the 
Director‑General to the Commission on the performance of his duties as Authorising Officer by delegation, in the 
management of the EU budget. This report is also communicated to the European Parliament and the Council. It can 
be found at: http://ec.europa.eu/atwork/key‑documents/index_en.htm

Article 318 evaluation report: Article 318 of the Treaty on the Functioning of the European Union requires the 
Commission to ‘submit to the European Parliament and to the Council an evaluative report on the Union’s finances 
based on the results achieved’. The report can be found at: http://ec.europa.eu/smart‑regulation/evaluation/
documents_en.htm

Budget support: This method of aid delivery involves transferring funds from an external financing agency to the 
partner government’s national treasury. The funds thus transferred are managed in accordance with the recipient’s 
budgetary procedures. Budget support programmes usually include capacity development measures to help 
partner countries strengthen their policymaking and management, as well as to promote dialogue on the design, 
implementation and results of national and sectorial policies.

External Assistance Management Report: EU delegations’ main reporting document to EuropeAid. It has to be 
made available to the European Parliament and the Council.

European Investment Bank (EIB): The EIB is the EU’s bank. It is owned by and represents the interests of the EU 
Member States. It works closely with other EU institutions to implement EU policy. It is the largest multilateral 
borrower and lender by volume in the world. More than 90 % of its activity is focused on Europe, but it also 
supports the EU’s external and development policies.

European Development Funds (EDFs): The EDFs are the main instrument for providing EU aid for development 
cooperation to African, Caribbean and Pacific (ACP) States and overseas countries and territories (OCTs). The 
partnership agreement signed in Cotonou on 23 June 2000 for a period of 20 years is the current framework for 
the EU’s relations with ACP States and OCTs. Its main focus is on reducing and eventually eradicating poverty. 
The EDFs are funded by the EU Member States, governed by their own financial regulations and managed, by the 
Commission, outside the EU general budget.

European External Action Service (EEAS): The EEAS was launched in December 2010 under the Treaty of Lisbon. 
Among its activities, it prepares country and regional allocations and EU cooperation strategies jointly with relevant 
Commission departments, as part of the programming cycle for most external action instruments.

Glossary and abbreviations 

05

EuropeAid: Within the European Commission, the Directorate‑General for Development and Cooperation ‑  
EuropeAid is responsible for:
—  formulating EU development policy and defining sectoral policies in the field of external aid;
—  drawing up the multiannual programming of the external aid instruments together with the European External 

Action Service (EEAS);

—  fostering coordination between the EU and the Member States on development cooperation and externally 

representing the EU in this field.

Evaluation findings: For the purpose of this report, this term covers conclusions, recommendations and lessons 
learned from an evaluation.

EU delegations: The EU is represented through 139 EU delegations and offices around the world which present, 
explain and implement EU policies (e.g. the development cooperation policy), analyse and report on policies and 
developments of the countries to which they are accredited, and conduct negotiations in accordance with a given 
mandate.

Internal Audit Capability: is a unit of a Commission’s directorate‑general which reports directly to the 
director‑general. Its task is to provide independent assurance on the effectiveness of the internal control system 
with a view to improving the directorate‑general’s operations.

Financing instruments: These are regulations, usually issued by the European Parliament and the Council. They 
provide the financing needed to support an EU policy. The EU’s development policy is supported by several 
financing instruments.

Financial perspectives: A multiannual financial planning framework which sets limits on EU expenditure. It puts 
the priorities set for the Union’s policies into financial terms and is at the same time an instrument of budgetary 
discipline and planning. The current financial perspectives cover the period from 2014 to 2020; the previous one 
covered the period from 2007 to 2013.

OECD‑DAC: Development Assistance Committee. A committee within the OECD (Organisation for Economic 
Cooperation and Development) dealing specifically with development assistance.

Result: This term covers:
—  outputs: that is produced or accomplished with the resources allocated to an intervention (e.g. grants 

distributed to farmers, training courses delivered to unemployed people, road built);

—  outcomes: the immediate changes resulting from an intervention (e.g. improved accessibility to an area due to 

the construction of a road, trainees who have found employment); and

—  impacts: longer‑term socioeconomic consequences, direct or indirect, positive or negative, that can be 

observed a certain period after the completion of an intervention.

06

V
The evaluation and ROM systems do not ensure that 
maximum use is made of findings because proper 
mechanisms are not in place to monitor their fol‑
low‑up and dissemination.

VI
The evaluation and ROM systems do not provide 
adequate information on results achieved due to 
insufficiently well‑defined objectives and indica‑
tors, the limited proportion of ex post evaluations 
and ROMs and inherent limitations in the evaluation 
methodology for budget support. These factors limit 
considerably EuropeAid’s capacity to account for the 
actual results achieved.

VII
The Court provides recommendations on the efficient 
use of evaluation and ROM resources, the prioritisation 
and monitoring of evaluations, the implementation 
of quality control procedures, the demonstration of 
results achieved and the follow‑up and dissemination 
of evaluation and ROM findings.

Executive  
summary

I
Within the Commission’s decentralised organisational 
framework, the Directorate‑General for Development 
and Cooperation — EuropeAid (EuropeAid) has set 
up its own results accountability framework which 
comprises the monitoring, evaluation and reporting 
of its activities. This report examines two of the key 
elements of this framework, namely evaluations and 
results oriented monitoring (ROM). Evaluation is the 
systematic and objective assessment of an on‑ 
going or completed programme or policy, its design, 
implementation and results. ROM is a standardised 
external review, specific to external aid, designed to 
look at programmes’ performance. The main purposes 
of these parts of the accountability framework are to 
improve the implementation of ongoing programmes 
and the design of future programmes and policies 
through feedback and lessons learned, and to provide 
a basis for accountability.

II
The Court found that EuropeAid’s evaluation and ROM 
systems are not sufficiently reliable.

III
Overall, EuropeAid’s evaluation and ROM functions are 
well organised but there is a lack of overall supervision 
of programme evaluation activities. Also, insufficient 
attention is paid to the efficient use of evaluation and 
ROM resources.

IV
The evaluation and ROM systems do not sufficiently 
ensure that relevant and robust findings are produced. 
Programme evaluation plans are based on insuffi‑
ciently clear prioritisation criteria. There is no monitor‑
ing system to identify and address frequent deviations 
from evaluation plans. Quality control procedures are 
not implemented consistently for ROM and pro‑
gramme evaluations.

Introduction

07

01 
In the current economic and political 
context, there is an increased need 
for the Commission to ensure and 
demonstrate that EU funds are used in 
an economic, efficient and effective 
manner. The European Parliament and 
the Council have indicated a need for 
a clearer view of the results achieved1 
with regard to the EU’s main objec‑
tives. Information on results achieved 
can only be obtained with a results 
accountability framework comprising 
three elements: monitoring, evaluation 
and reporting.

The Commission’s 
evaluation framework

02 
Evaluation is the systematic and ob‑
jective assessment of an ongoing or 
completed programme or policy, its 
design, implementation and results. 
The main purposes are to ascertain 
whether the objectives have been 
met and to formulate recommenda‑
tions with a view to improving future 
interventions so as to support deci‑
sion‑making by the Commission, the 
European Parliament and the Council.

03 
Evaluation is essential for planning, 
designing and implementing EU poli‑
cies and interventions, and enhancing 
transparency and democratic account‑
ability. This is emphasised by Article 
318 of the Treaty on the Functioning 
of the European Union, which requires 
the Commission to submit annually 
to the European Parliament and the 
Council an evaluation report on the 
Union’s finances on the basis of the 
results achieved (Article 318 evalua‑
tion report). As regards the first three 
reports produced by the Commission, 
in February 2014, the European Parlia‑
ment pointed out2 that they did not 
provide the budgetary authority with 
a clear view of the real extent to which 
the Union’s main objectives have been 
achieved.

04 
The Commission’s evaluation frame‑
work is set out in a number of policy 
and regulatory documents:

(a)  the financial regulations applicable 

to the EU budget3;

(b)  the communication on reinforcing 

the use of evaluation4;

(c)  the communication on strategic 

evaluation5; and

(d)  the communication on strengthen‑
ing the foundation of Smart Regu‑
lation — improving evaluation6.

1 

For instance, Council 
recommendations to the 
Commission in relation to the 
2011 discharge on ‘getting 
results’ (document 5752/13 
ADD1, p. 31). European 
Parliament’s 
recommendations in the 
context of the discharge 2011, 
see recommendations 248 for 
the EEAS and EuropeAid, 308 
and 309 for the Commission as 
a whole (document 
P7_TA‑PROV(2013)0122).

2  Resolution of 26 February 2014 
the evaluation of the Union’s 
finances based on the results 
achieved: a new tool for the 
European Commission’s 
improved discharge 
procedure. Document 
P7_TA(2014)0134.

3  Article 30(4) of Regulation (EU, 

Euratom) No 966/2012 of the 
European Parliament and of 
the Council of 25 October 2012 
on the financial rules 
applicable to the general 
budget of the Union (OJ L 298, 
26.10.2012, p. 1).

4 

SEC(2007) 213 of 
21 February 2007: 
Communication to the 
Commission from 
Ms Grybauskaité in agreement 
with the President: 
‘Responding to Strategic 
Needs: Reinforcing the use of 
evaluation’.

5  C(2001) 3661 of 

12 November 2001: 
Communication of the 
President ‘Putting evaluation 
into practice within the 
Commission’.

6  COM(2013) 686 final of 

2 October 2013 ‘Strengthening 
the foundation of Smart 
Regulation — improving 
evaluation’.

Introduction

05 
Individual directorates‑general are 
responsible for evaluating their own 
activities and can structure their 
evaluation systems to suit their needs 
and requirements, in compliance with 
Commission‑defined standards con‑
cerning the organisation, the selection, 
the design and the use of evaluations 
findings.

EuropeAid’s evaluation 
and ROM systems

06 
EuropeAid’s evaluation and results 
oriented monitoring (ROM) systems 
are part of its results accountability 
framework which is based on the pol‑
icy and regulatory documents listed 
in paragraph 4 as well as in policy and 
regulatory documents more specific to 
the EU’s development aid:

(a)  the financial regulations applicable 
to the EDF7 and the other financing 
instruments supporting the EU’s 
development policy;

(b)  the Paris Declaration, the Accra 

Agenda for Action8 and the Busan 
Partnership Agreement9;

(c)  the European Consensus on 

Development10;

(d)  the communication on the ‘Agen‑

da for Change’11.

07 
EuropeAid manages two types of 
evaluations, which are carried out by 
external contractors:

(a)  strategic evaluations, which 

comprise geographic evaluations 
(covering a country or region) and 
thematic evaluations, which cover 
a sector (e.g. education, food se‑
curity) or a method of aid delivery 
(e.g. budget support, cooperation 
with development banks); and

(b)  programme12 evaluations, that can 
be carried out during the imple‑
mentation (mid‑term evaluation), 
on completion (final evaluation) or 
after completion (ex post evalua‑
tion) of an intervention.

08 
These evaluations assess performance 
against five criteria developed by the 
Development Assistance Committee of 
the Organisation for Economic Coop‑
eration and Development (OECD‑DAC), 
along with the international donor 
community13: relevance, effectiveness, 
efficiency, sustainability and impact. 
In addition, evaluations have to assess 
the EU’s comparative advantage and 
the coherence with other EU policies 
and programmes, other donors’ ac‑
tions and partner countries’ policies.

08

7 

10th EDF: Articles 12 and 27 of 
Council Regulation (EC) 
No 215/2008 of 
18 February 2008 on the 
Financial Regulation 
applicable to the 10th 
European Development Fund 
(OJ L 78, 19.3.2008, p. 1).

11th EDF: Decision No 1/2013 
of the ACP‑EU Council of 
Ministers of 7 June 2013 
adopting a protocol on the 
multiannual financial 
framework for the period 
2014–20 under the Partnership 
Agreement between the 
members of the African, 
Caribbean and Pacific Group 
of States of the one part, and 
the European Community and 
its Member States, of the other 
part (OJ L 173, 26.6.2013, p. 67).

8  Paris Declaration on Aid 

Effectiveness, 2005 and Accra 
Agenda for Action, 2008: 
http://www.oecd.org/dac/
effectiveness/parisdeclaratio‑
nandaccraagendaforaction.
htm

9  Busan Partnership Agreement 
for Effective Development 
Cooperation, 2011: http://
www.oecd.org/dac/
effectiveness/fourthhighlevel‑
forumonaideffectiveness.htm.

10  Joint statement by the Council 
and the representatives of the 
governments of the Member 
States meeting within the 
Council, the European 
Parliament and the 
Commission on European 
Union Development Policy: 
‘The European Consensus’ 
(OJ C 46, 24.2.2006, p. 1).

11  COM(2011) 637 final of 

13 October 2011 ‘Increasing 
the impact of EU 
Development Policy: an 
Agenda for Change’.

12  The term programme used in 
this report refers to projects, 
which are small‑scale 
interventions, and 
programmes, which are 
large‑scale interventions.

13  http://www.oecd.org/dac/

evaluation/daccriteriaforevalu‑
atingdevelopmentassistance.
htm

 
09

14  EAMRs were required annually 

from 2014.

Introduction

09 
The ROM system is used by EuropeAid 
to monitor, on the spot, the qual‑
ity of its programmes. It is based on 
short on‑site visits to programmes by 
external experts who rate the pro‑
gramme’s performance (see Annex I) 
in respect of the above mentioned 
five OECD‑DAC evaluation criteria 
(see paragraph 8). The grading aims 
to provide a comparison between 
different programmes and for the ag‑
gregation of overall portfolio perfor‑
mance data. Compared to evaluations, 
ROM is a lighter tool which provides 
a quick overview of the performance 
of programmes. Like programme 
evaluations, they can be carried out 
during or after the implementation of 
programmes.

10 
EuropeAid produces the following key 
reports:

(a)  the half yearly External Assistance 
Management Report (EAMR), 
EU Delegations’ main reporting 
document14;

(b)  the reports of EuropeAid’s direc‑

tors about the management of the 
funds delegated to them;

(c)  the Annual Activity Report (AAR) 

submitted by EuropeAid’s Director 
General to the Commissioner as 
well as to the European Parliament 
and the Council;

(d)  the Annual Report (AR) on the 

European Union’s development 
and external assistance policies 
submitted by the Commission to 
the European Parliament and the 
Council and;

(e)  the contribution to the Article 318 
evaluation report submitted by 
the Commission to the European 
Parliament and the Council.

10

15  Evaluation; Quality and 

Results; Planning ACP and 
Horizontal Coordination; 
Geographical Coordination 
Asia and Pacific; Education, 
Health, Research, Culture; 
Rural Development, Food 
Security, Nutrition; Africa‑EU 
Partnership, Peace Facility; 
Regional Programmes 
Neighbourhood South.

Audit scope  
and approach

11 
The Court examined whether Europe‑
Aid has reliable evaluation and ROM 
systems. The audit focused on three 
main questions:

(a)  Are EuropeAid’s evaluation and 
ROM functions appropriately or‑
ganised and resourced?

13 
The audit was carried out between 
November 2013 and March 2014 and 
included:

(a)  a review of EuropeAid’s evaluation 
and ROM guidelines and of the 
main documents and processes in 
the management of evaluations;

(b)  Do EuropeAid’s evaluation and 

(b)  interviews with officials from eight 

ROM systems ensure the provision 
of relevant and robust findings?

units at EuropeAid15;

(c)  Do the evaluation and ROM sys‑

tems ensure the maximum use of 
the findings made?

12 
The audit covered the evaluation and 
ROM systems in place in the period 
from 2007 to 2013. Where adequate 
information was available and could 
be checked by the Court, it took ac‑
count of changes adopted or planned 
by EuropeAid for the period from 
2014 to 2020.

(c)  visits to the EU delegations in Be‑

nin and India, involving interviews 
with the delegations staff and a re‑
view of relevant documentation;

(d)  a review of programming docu‑

ments (see Annex II), programme 
financing agreements (see An-
nex III), strategic and programme 
evaluations (see Annex IV), and 
ROM reports (see Annex V); and

(e)  a survey sent to 50 EU delegations 
to which 41 answers were received 
(see Annex VI).

Observations

11

16  There are five ROM 

coordinators in total: one for 
geographical programmes in 
a region (Africa — Caribbean 
— Pacific, Asia, Latin America, 
Neighbourhood) and one for 
thematic programmes.

(b)  the Quality and Results unit super‑
vises the ROM system, including 
overall quality assurance com‑
mon to the whole organisation, 
the coherence between ROM 
and programme evaluations, and 
contributes to the analysis of ROM 
and programme evaluation data. 
It provides the ROM aggregated 
performance ratings for the AAR 
and AR;

(c)  ROM coordinators at EuropeAid16 

supervise the ROM system for their 
respective areas of intervention. In 
EuropeAid’s operational units and 
EU delegations, ROM focal points 
provide support to the experts in 
organising field visits.

16 
The Evaluation unit and the Qual‑
ity and Results unit report directly 
to a deputy director‑general. This is 
a sufficiently senior position in the or‑
ganisation to ensure that they receive 
support from senior levels of manage‑
ment for their roles.

Overall, the evaluation 
and ROM functions are 
well organised but 
EuropeAid could pay 
more attention to the 
efficient use of resources

14 
The Court examined whether the 
evaluation and ROM systems provide 
for a sound distribution of responsi‑
bilities and ensure the efficient use of 
financial and staff resources.

There is a sound distribution 
of responsibilities but 
insufficient supervision of 
programme evaluations

15 
EuropeAid’s organisation successfully 
combines centralised and decentral‑
ised responsibilities. As regards cen‑
trally managed responsibilities:

(a)  the Evaluation unit manages stra‑
tegic evaluations, which requires 
it to have a comprehensive view 
of EuropeAid’s activities. This unit 
is responsible for developing the 
evaluation methodology and for 
coordinating and monitoring Euro‑
peAid’s evaluation activities. It also 
has a central role in the reporting 
of strategic evaluations findings in 
the AAR and AR and in EuropeAid’s 
contribution to the Commission’s 
Article 318 evaluation report;

12

17  EuropeAid estimates that this 
annual cost will increase to 
about 25,5 million euro, due in 
particular to longer on‑site 
visits (from an average of 
2 days to 8 days) and the use 
of more senior experts.

18  Common Relex Information 

System.

Observations

17 
The management of programme eval‑
uations is decentralised to EuropeAid’s 
operational units and EU delegations. 
This has potential benefits in terms 
of ensuring the take‑up of evaluation 
results. However, programme man‑
agers have a considerable degree of 
autonomy in managing programme 
evaluations as they are responsible 
at all stages of the process from the 
initiation of the evaluations until the 
follow‑up of results. In the absence of 
overall coordination and monitoring of 
this activity EuropeAid has no over‑
view and there is little supervision by 
senior management.

EuropeAid does not give 
enough consideration to the 
efficient use of resources

18 
The Commission’s evaluation stand‑
ards require that evaluation activities 
are appropriately resourced to meet 
their purposes. In particular, each 
directorate‑general must ensure that 
staff and financial resources are clearly 
identified and proportionately al‑
located for evaluation activities to be 
carried out.

19 
EuropeAid has insufficient information 
on the resources used by its evalua‑
tion and ROM systems. Information is 
available as regards the annual cost 
of strategic evaluations, which ranges 
between 5 and 6 million euro, and 
of ROM, which amounted to 17 mil‑
lion euro over the 2007–13 period17. 
According to estimates made in 
2010 and 2013 by EuropeAid, the 
annual cost of programme evalua‑
tions ranges between 12 and 15 mil‑
lion euro. The total annual cost of 
evaluations and ROM would thus 
amount to 34–38 million euro. Howev ‑
er, as EuropeAid has no reliable infor‑
mation on the number of programme 
evaluations and since their cost cannot 
be easily identified in the IT manage‑
ment information system18, the Court 
cannot confirm the accuracy of these 
estimates.

20 
There are 16 staff in the Evaluation unit 
and about one full‑time equivalent 
staff member involved in ROM at the 
Quality and Results unit. But there is 
no management information on staff 
resources involved in programme 
evaluations and ROM reviews at the 
level of programme managers, ROM 
focal points and ROM coordinators. On 
the basis of its survey, the Court esti‑
mates that there are approximately 60 
full time equivalent staff dealing with 
ROM and programme evaluations in 
the EU delegations. The total number 
of full‑time equivalent staff dealing 
with evaluations and ROM is thus likely 
to be well above 80.

Observations

13

21 
Due to the lack of reliable information 
EuropeAid cannot ensure that staff 
and financial resources are appropriate 
and proportionately allocated to the 
various evaluation and ROM activities.

EuropeAid’s evaluation 
and ROM systems do not 
sufficiently ensure 
relevant and robust 
findings

19  The only rule is that there 

should not be a ROM the same 
year a programme evaluation 
is carried out.

22 
The Court found inadequate coordina‑
tion between programme evaluations 
and ROMs, which is likely to lead to 
inefficiencies. Programme evaluations 
and ROMs follow completely different 
planning approaches and criteria19 (see 
paragraphs 26 and 30). Programmes 
can be over‑ or under‑assessed due 
to an inappropriate combination of 
ROM and evaluations, or sometimes 
not assessed at all, as is frequently the 
case at the post completion stage (see 
paragraphs 43 to 46). This is confirmed 
by the Court’s survey: only half of 
the replies consider that programme 
evaluations and ROMs complement 
each other well.

23 
The Court examined whether evalu‑ 
ations and ROMs were selected on the 
basis of appropriate criteria and are 
carried out on time. Furthermore, it 
examined whether they were carried 
out by sufficiently qualified experts, 
whether quality control procedures 
were applied effectively and whether 
they provided sufficient evidence on 
the results achieved by EuropeAid’s 
interventions.

There are some weaknesses 
in the selection of 
programme evaluations 
and timely delivery of 
evaluations

Multiannual plans for strategic 
evaluations are based on 
appropriate criteria

24 
The indicative multiannual evaluation 
plan for the 2007–13 period aimed 
to achieve full geographic coverage 
with a view to drawing lessons for the 
subsequent programming process. 
Thematic evaluations were selected 
on the basis of the materiality of funds 
allocated and the importance of sec‑
tors in country‑level and regional EU 
cooperation strategies. The objective 
was also to increase the number of 
evaluations of aid delivery methods 
(e.g. budget support, cooperation with 
the EIB, development banks, coopera‑
tion with UN organisations).

14

20  Article 17 of Regulation (EU) 

No 236/2014 of the European 
Parliament and of the Council 
of 11 March 2014 laying down 
common rules and procedures 
for the implementation of the 
Union’s instruments for 
financing external action 
(OJ L 77, 15.3.2014, p. 95).

21  Key policy priorities are:

•   Human rights, democracy, 
governance (including rule 
of law, gender equality and 
the empowerment of 
women, public sector 
management, tax policy 
and administration, 
corruption, civil society and 
local authorities, natural 
resources, 
development‑security);
•   Inclusive and sustainable 
growth (social protection, 
health, education and jobs, 
business environment, 
regional integration and 
world markets, sustainable 
agriculture and energy).

22  Country evaluations took on 
average 18 months instead of 
the planned 12 months.

Observations

25 
For the rolling 2014–18 work pro‑
gramme, the selection criteria were 
further refined. The objective was to 
provide sufficient material to report, 
in 2017, to the European Parliament 
and the Council on the results and 
impacts of the financial instruments, 
as required by the common rules and 
procedures for the implementation of 
the Union’s instruments for financing 
external action20. Thematic evaluations 
were selected on the basis of priority 
themes of the Agenda for Change21, 
and the aim was to ensure a propor‑
tional balance between regions (Africa, 
Latin America and Asia) and types of 
countries (middle‑income countries, 
fragile states and countries with sig‑
nificant budget support programmes).

The criteria for programme 
evaluation selection are unclear

26 
The main reason for carrying out 
a programme evaluation is that it is 
an obligation or a possibility set out 
in the financing agreement signed 
with the beneficiary or implement‑
ing partner. The provision for evalu‑ 
ations in these agreements is decided 
on a case‑by‑case basis at the design 
phase of the programme but no guid‑
ance is given on how to make this deci‑
sion. The Court’s review of 20 recent 
programmes (see Annex III) identified 
significant and unexplained differ‑
ences in approach:

(a)  mid‑term evaluations were provid‑
ed for in all programmes but were 
compulsory in only half of them;

(b)  ex post evaluations were provided 
for in only two thirds of the pro‑
grammes and were compulsory in 
only one of them.

Evaluations are often not 
delivered as planned

27 
EuropeAid has not set up a proper 
system to monitor the implementation 
of strategic and programme evaluation 
plans, analyse the reasons for devia‑
tions, and address the systemic causes 
of delay.

28 
Strategic evaluations often took much 
more time than planned22. This con‑
tributed to the incomplete execution 
of the indicative multiannual evalua‑
tion programme for the 2007–13 per‑ 
iod; only 70 % of the selected evalu‑
ations were completed. According to 
EuropeAid, this was particularly due to 
staff resource constraints in the Evalu‑
ation unit.

15

23  On average, 3,5 annual 

evaluations were planned by 
EU delegations, and 
2,9 evaluations were 
performed.

24  Mentioned as the main reason 
for not carrying out a planned 
evaluation.

25  The main reasons for 

departing from the standard 
criteria are: (i) to avoid 
unnecessary monitoring of 
well‑managed programmes or 
programmes evaluated 
recently; (ii) to cover 
problematic programmes 
which fall below the financial 
threshold; and (iii) to 
strengthen monitoring efforts 
on programmes with 
insufficient internal 
monitoring.

26  Special Report No 1/2011 ‘Has 

the devolution of the 
Commission’s management of 
external assistance from its 
headquarters to its 
delegations led to improved 
aid delivery’, paragraph 25 
(http://eca.europa.eu).

27  Terms of reference or technical 

specifications.

Observations

29 
Even though programme evaluations 
are usually carried out according to 
their timetable, the Court’s survey 
found that, in the period 2010–13, 
there was a significant difference 
between the evaluations planned 
and carried out23. The provisions set 
out in the financing agreements were 
not systematically applied during 
programme implementation as evalu‑ 
ations were sometimes postponed or 
cancelled. The Court’s survey indi‑
cates that EU delegations take such 
decisions on the basis of factors like 
limited staff resources24, the needs of 
evaluation users and problematic pro‑
gramme implementation. The reasons 
for not carrying out planned evalu‑
ations were not documented. This 
increases the risk that decisions not to 
carry out an evaluation are not taken 
for sound reasons, or at the appropri‑
ate level.

The ROM selection 
methodology reduces the 
usefulness of ROM

30 
The selection of ROM reviews is based 
on clearly defined standard criteria25 
and is made in consultation between 
the ROM contractors and programme 
managers. ROM reviews are usually 
carried out according to plan and 
reports are usually delivered without 
significant delays.

31 
However, in its Special Report 
No 1/201126 the Court pointed to key 
limitations which reduce the useful‑
ness of ROM as an indicator for the 
overall aid portfolio, notably that 
the sample of projects selected is 
not established on a statistical but 
rather on a judgemental basis. In 2011, 
EuropeAid’s Internal Audit Capabil‑
ity also found that, as the selection is 
not risk‑based, ROM reviews do not 
provide the feedback to programme 
managers where it is most needed to 
improve programme implementation.

The procedures to ensure 
the quality of programme 
evaluations and ROMs are 
implemented inconsistently

Contracting procedures ensure 
the recruitment of sufficiently 
qualified experts

32 
Evaluations are carried out by external 
consultants contracted by EuropeAid 
through public procurement proce‑
dures. Proposals submitted by consul‑
tancy firms are scored on the basis of 
specific requirements27, in particular 
those concerning the experts’ profes‑
sional qualification. Team leaders and 
senior experts usually have 10 to 15 
years of professional experience in 
the field of intervention, plus 5 years 
of experience in evaluation for team 
leaders.

36 
This was not the case for programme 
evaluations. According to the Court’s 
survey, only 39 % of EU delegations 
set up a reference group always or 
most of the time31, and a reference 
group was rarely formally set up by 
EuropeAid’s operational units and EU 
delegations visited by the Court. Some 
consultations may take place between 
the programme manager and other 
stakeholders at various stages of the 
evaluation process (e.g. for drafting 
the terms of reference or reviewing the 
draft evaluation report). However, in 
the absence of appropriate documen‑
tation, it was not possible for the Court 
to assess if this was frequent practice.

37 
EuropeAid’s evaluation guidelines 
provide that the quality of the draft 
and final evaluation reports must be 
assessed on the basis of nine well‑de‑
fined criteria32. For this purpose, 
a quality grid must be filled by two 
persons who discuss the nine criteria. 
For strategic evaluations, the qual‑
ity grid was filled systematically and 
published together with the evalu‑ 
ation report on EuropeAid’s website33. 
For programme evaluations, according 
to the Court’s survey, only 71 % of EU 
delegations assess the quality of the 
evaluation report and only 37 % fill the 
assessment grid34.

16

28  There are five ROM 

contractors in total: one per 
region for geographical 
programmes (Africa — 
Caribbean — Pacific; Asia; 
Latin America; 
Neighbourhood) and one for 
the thematic programmes.

29  Also referred to as ‘steering 

group’ in the Commission.

30  Depending on the type and 
place of the evaluation, the 
reference groups may include 
representatives from 
EuropeAid, the EEAS, other 
Commission’s departments, 
partner country’s authorities, 
the civil society organisations, 
other donor agencies, or other 
experts.

31  No formal reference groups 

were created in the 
delegations and operational 
units visited during the audit 
except in the case of 
programme evaluations 
similar to strategic evaluations.

32  The nine criteria are: (1) 

Meeting needs; (2) Relevant 
scope; (3) Defensible design; 
(4) Reliable data; (5) Sound 
analysis; (6) Credible findings; 
(7) Valid conclusions; (8) Useful 
recommendations; (9) Clear 
report. For programme 
evaluations, criteria 2 and 3 are 
combined into one: 
Appropriate design. Each 
criteria is rated by five levels: 
excellent, very good, good, 
poor and unacceptable. 
(http://ec.europa.eu/
europeaid/evaluation/
methodology/guidelines/
gui_qal_flr_en.htm#03).

33  http://ec.europa.eu/

europeaid/evaluation‑ 
reports_en

34  This grid was filled in for only 

one out of the ten programme 
evaluations reviewed by the 
Court, the African Peace 
Facility evaluation.

Observations

33 
ROM reviews are also carried out by 
external consultancy firms (ROM con‑
tractors) selected through public pro‑
curement procedures28. The required 
experience of development coopera‑
tion in the given region is 10 years 
for senior experts, 5 to 10 years for 
mid‑level experts and up to 5 years 
for junior experts. For contracts to be 
signed in 2014, EuropeAid required 
ROM contractors to hire specialised 
experts on long‑term assignments in 
order to further enhance the level of 
expertise.

Quality control procedures for 
programme evaluations and 
ROMs are not systematically 
applied

34 
The Commission’s evaluations stand‑
ards stipulate that, save in duly justi‑
fied cases, a reference group29 must 
be set up for each evaluation to advise 
on the terms of reference, support 
the evaluation work and take part in 
assessing the quality of the evaluation 
with appropriate regularity. Europe‑
Aid’s evaluation guidelines stress that 
quality assurance at each step of the 
process is a key task of the evaluation 
manager, together with the reference 
group, which should include repre‑
sentatives of the main stakeholders 
concerned30.

35 
Reference groups are systematically 
set up for strategic evaluations. They 
meet on a regular basis and their work, 
including quality review at all stages, is 
documented in meeting minutes.

Observations

38 
Programme managers are required to 
assess the quality of ROM reports by 
filling a compulsory response sheet. 
This was done for less than 60 % of 
ROM reports delivered over the period 
from 2011 to 2013.

The evaluation and ROM 
systems do not provide 
adequate information on 
results achieved

Programme objectives and 
indicators are not well defined

39 
To enable an evidence‑based assess‑
ment of achievements to be carried 
out, there must be specific, measur‑
able and time‑bound objectives for 
programmes with objectively verifi‑
able indicators. The objectives men‑
tioned in high level policy documents 
and legal texts establishing the financ‑
ing instruments are prepared in a po‑
litical context and, while they should 
provide a basis, are often not focused 
enough to be useful at implementing 
level. They need to be translated into 
more specific operational objectives 
when programming aid for a country, 
region or sector, and designing indi‑
vidual interventions.

40 
Out of the 15 thematic, regional and 
country‑level programming docu‑
ments reviewed by the Court (see 
Annex II), five35 contained clearly 
described objectives and indicators, 
but set no baselines or targets against 
which to assess performance. For the 
other 10, objectives were unclear and 
indicators often lacking (see Box 1).

41 
Concerning individual programmes, 
several Court’s reports36 have high‑
lighted that objectives are insuffi‑
ciently clear and that no measurable 
indicators are defined. Out of the 10 
programme evaluation reports re‑
viewed by the Court, four mentioned 
unclear objectives and indicators, as 
well as the absence of baselines and 
unavailability of monitoring data, as 
major weaknesses37 (see Box 2). Only 
two evaluations made positive com‑
ments in this respect38.

42 
In the absence of clearly defined 
objectives and indicators, it is not 
possible for an evaluation to establish 
evidence‑based performance assess‑
ments. This issue was also mentioned 
in a 2011 study commissioned by 
EuropeAid39.

17

35  Food Security, Benin, Chad, 
Uruguay and Zambia.

36  E.g. Special Report No 1/2012 
‘Effectiveness of European 
Union development aid for 
food security in sub‑Saharan 
Africa’, paragraph 51 and 
Special Report No 13/2013 ‘EU 
Development Assistance to 
Central Asia’, paragraph 52. 
(http://eca.europa.eu).

37  Similar comments were made 
by two ROM contractors, one 
responsible for the ROM, for 
the European Neighbourhood 
Partnership Instrument and 
the other responsible for the 
ROM for the Centrally 
Managed Thematic 
Programmes, in their annual 
reports 2011, 2012 and 2013.

38  Benin: PACTE — programme 
d’appui aux collectivités 
territoriales ; Food Facility 
programmes implemented 
with the African Union 
— Interafrican Bureau for 
Animal Resources (AU‑IBAR): 
Livestock Emergency 
Intervention to mitigate Food 
Crisis in Somalia (LEISOM) and 
Vaccines for the Control of 
Neglected Animal Diseases in 
Africa (VACNADA).

39  Study on Legal Instruments 

and Lessons Learned from the 
Evaluations Managed by the 
Joint Evaluation Unit, July 
2011, paragraph 26, p. ix (it 
does not cover the EDF). 
http://ec.europa.eu/
europeaid/
evaluation‑reports_en, under 
the section ‘per instrument / 
channel’.

Observations

18

1
x
o
B

Examples of vague objectives in multiannual programming documents

Investing in People: Mid-term Review of Strategy Paper for Thematic Programme (2007–13) and  
Multiannual Indicative Programming 2011–13

As regards the strengthening of health systems and universal access to basic health care:

‘In terms of delivery, the thematic action should focus on supporting activities that:

 ο stimulate effective leverage of resources for inclusive health system strengthening and universal coverage 

of health service delivery;

 ο promote and support the application of innovative ways and good practices to address weaknesses in 

health service delivery and to improve quality of care;

 ο support the role of civil society, professional associations and other stakeholders in their advocacy, policy 

monitoring and advisory role in the area of health;

 ο support platforms for policy dialogue and exchange of good practices on different aspects of health at 

regional and global levels;

 ο support partner countries in the process of implementing international legal commitments in the area of 

health.’

CSP/NIP Bolivia 2007–13

‘Strengthening the productive sector comprising micro‑enterprises and SMEs, and improving the institutions 
framework in order to increase their productivity and competitiveness, which will lead to better market access 
and integration with the purpose of creating decent and sustainable work.’

2
x
o
B

Examples of vague objectives in programmes

EuroMed Migration II 2008–11

‘Strengthen cooperation in the management or migration so as to build the MEDA partners’ capacities to pro‑
vide an effective, targeted and comprehensive solution for the various forms of migration.’

African Peace Facility 2003–13

‘Support the operationalisation of the African Peace and Security Architecture (APSA) and to increase the 
effectiveness of the African Union and the African regional organisations on the conduct of peace support 
operations in Africa.’

 
 
19

40  See ROM Handbook 2012, 

p. 57.

Observations

Evaluations and ROMs focus 
more on the implementation 
of programmes than on results 
achieved

43 
EuropeAid’s evaluation guidelines 
indicate that:

(a)  mid‑term and final evaluations 

are intended to draw lessons from 
the first years of the intervention 
implementation, adjust the ongo‑
ing intervention where necessary 
and provide an analysis of the first 
results and impacts achieved;

(b)  ex post evaluations are mainly 

concerned with checking achieved 
impacts and assessing the sustain‑
ability of the intervention’s ben‑
efit, report to the institutions that 
have allocated the resources and 
help to transfer acquired experi‑
ence to other countries or sectors.

44 
Ex post programme evaluations are 
seldom carried out. For the two EU 
delegations visited, only three out of 
the 17 evaluations carried out in the 
2010–13 period were ex post evalu‑ 
ations. Financing agreements rarely set 
an obligation to perform such evalu‑
ations (see paragraph 26(b)). Another 
reason for the lack of ex post evalu‑ 
ations is that programme evaluations 
are funded from the programme’s own 
budget and are therefore constrained 
by its completion date.

45 
According to EuropeAid, additional 
contributing factors are the una‑
vailability of monitoring data after 
programme completion and the 
possible evolutions in the partner 
country’s context that may affect the 
programme’s effectiveness. The Court 
does not consider these to be valid 
arguments: the monitoring system 
can be designed to continue collect‑
ing the data deemed necessary during 
a certain period after programme 
completion (see paragraph 47) and 
the purpose of evaluation is also to 
identify and assess the impact of unex‑
pected factors.

46 
The ROM guidelines40 indicate that ex 
post ROMs allow solid information to 
be gathered about a project’s design, 
real impact and sustainability which 
can only be fully appreciated after 
a project has ended. But ex post ROMs 
represented on average only 11 % of 
the ROMs carried out in the 2008–12 
period. EuropeAid decided to discon‑
tinue with them from 2014. It consid‑
ered that the methodology for ex post 
ROM was inappropriate for assessing 
impact and that the revised ROM sys‑
tem aims only to support programme 
management and implementation.

20

41  http://www.oecd.org/dac/
evaluation/dcdndep/
Methodological%20
approach%20BS%20
evaluations%20Sept%20
2012%20_with%20cover%20
Thi.pdf

42  EuropeAid has managed four 
budget support evaluations: 
Mali and Tunisia (2011), 
Tanzania and South Africa 
(2013).

43  AAR 2011, AAR 2013 and AR 

2012.

44  Annual Report of the Court of 

Auditors on the 
implementation of the budget 
concerning the financial year 
2011, paragraph 10.12 (OJ 
C 344, 12.11.2012); Special 
Report No 1/2011, 
paragraph 59; Special Report 
No 13/2013, paragraphs 78 to 
80 (http://eca.europa.eu).

45  Annual Report of the Court of 

Auditors on the 
implementation of the budget 
concerning the financial year 
2012, paragraphs 10.21 and 
10.22 (OJ C 331, 14.11.2013).

Observations

47 
In contrast with the limited propor‑
tion of ex post programme evaluations 
and ROMs, the Results Measurement 
framework (ReM) implemented at the 
European Investment Bank (EIB) since 
2012 includes ex post monitoring until 
3 years after project completion and 
the EIB intends to use the data thus 
collected for ex post evaluations.

The methodology for 
evaluating budget support has 
inherent limitations

48 
Under the lead of EuropeAid’s Evalu‑
ation unit, the OECD DAC Network on 
Development Evaluation produced 
a document called ‘Evaluating Budget 
Support — Methodological Approach’ 
which was published in 201241. It is 
a common evaluation framework to 
evaluate budget support operations 
and provides guidelines on how to or‑
ganise and carry out these evaluations. 
The document recognises that in most 
cases, it will be difficult to quantify the 
budget support contribution, which 
will need to be assessed in a qualita‑
tive manner.

49 
While qualitative assessments are, of 
course, considered in all evaluations, 
the difficulty in establishing unam‑
biguous logical cause‑effect links 
between results observed and the 
budget support programmes weakens 
the reliability of evidence provided by 
the evaluation. This inherent limitation 
is not mentioned either in budget sup‑
port evaluation reports42 or in Europe‑
Aid’s reports43.

The reporting does not provide 
much evidence on results 
achieved

50 
EuropeAid’s capacity to assess and re‑
port on results achieved is significantly 
reduced by insufficiently clear objec‑
tives and indicators set for operational 
programmes and by the focus put on 
evaluations and ROM during the im‑
plementation of programmes.

51 
As the Court mentioned in several 
reports44, there is little information on 
the actual results of the aid in the EU 
delegations’ EAMRs and EuropeAid’s 
two key annual reports which tend to 
focus more on activities implemented, 
rather than on policy achievements.

52 
This is also the case with EuropeAid’s 
contribution to the Commission’s 
Article 318 evaluation reports. In its 
2012 annual report45, the Court found 
that, despite some positive develop‑
ments, these reports did not provide 
sufficient, relevant and reliable evi‑
dence on what the EU’s policies have 
achieved and that they were therefore 
not suitable for their intended use in 
the discharge procedure.

21

46  Called ‘fiche contradictoire’.

47  For example, the thematic 
evaluation of the European 
Commission support to 
respect of Human Rights and 
Fundamental Freedoms was 
issued in 2011. Many actions 
were to be completed after 
2012. This was the case for 
most evaluations completed 
in 2012 and 2013, for which 
recommendations were to be 
implemented mostly during 
the programming period 
2014–20.

48  European Union (2012), DAC 
Peer Review: Main Findings 
and Recommendations 
http://www.oecd.org/dac/
peer_reviews/europeanunion‑
2012dacpeerreviewmainfind‑
ingsandrecommendations.
htm

55 
The follow‑up of the action plan by 
the Evaluation unit is limited to 1 year 
after the publication of the evalua‑
tion report. This is often too short to 
implement actions plans which involve 
amendments to sectorial policies and 
country or regional EU cooperation 
strategies47.

56 
In 2012, an OECD DAC Peer Review48 
also found that EuropeAid’s Evaluation 
unit did not have a clear overview of 
the extent to which recommendations 
made on the strength of strategic 
evaluations were accepted and used to 
inform new policies and programmes. 
In 2014, a study commissioned by the 
Evaluation unit on the uptake of strate‑
gic evaluation results found that they 
were not used effectively.

Observations

EuropeAid’s evaluation 
and ROM systems do not 
ensure the maximum use 
of the findings made

53 
The Court examined whether Euro‑
peAid’s evaluation and ROM systems 
ensured that the findings made were 
properly followed up and disseminat‑
ed to all intended users.

Proper follow‑up 
mechanisms are not in place

There are weaknesses in 
the follow‑up of strategic 
evaluations findings

54 
Actions to be taken following a stra‑
tegic evaluation are described in 
a document46 published together with 
the evaluation report on EuropeAid’s 
website. While this is sound practice, 
most of such documents reviewed by 
the Court did not provide sufficiently 
detailed information on the planned 
actions, their timetable and the alloca‑
tion of responsibilities.

22

49  For six of the ten programme 
evaluations and six out of the 
eight ROM reviews examined.

50  AAR 2013, pp. 117 to 119 and 
150 to 162, and Annex 9.

51  Annual report on the 
European Union’s 
development and external 
assistance policies and their 
implementation in 2012, 
pp. 166 to 173.

52  COM(2013) 461 final of 

26 June 2013, Report from the 
Commission to the European 
Parliament and the Council on 
the evaluation of the Union’s 
finances based on the results 
achieved, pp. 22 to 23.

Observations

There is a lack of documented 
follow‑up for programme 
evaluations and ROM findings

Evaluation and ROM findings 
are well disseminated except 
for programme evaluations

57 
Programme evaluation and ROM find‑
ings are usually discussed within op‑
erational units or EU delegations and 
programme managers are responsible 
for ensuring that appropriate actions 
are taken. Unlike for strategic evalu‑
ations, and as the Court found in the 
two EU delegations visited, documen‑ 
ted action plans are not systematically 
prepared and monitored. According 
to the Court’s survey, only 44 % of EU 
delegations usually draw up action 
plans. Concerning ROM reviews, the 
response sheets reviewed by the Court 
described actions to be taken. But 
response sheets were completed for 
only 60 % of ROM reviews carried out 
in the period from 2011 to 2013.

58 
In more than half of the cases re‑
viewed49, the Court found evidence of 
use being made of some findings of 
the programme evaluations and ROMs. 
In the absence of an appropriate 
follow‑up system, EuropeAid cannot 
ensure that findings and recommenda‑
tions from programme evaluations and 
ROM reviews are used effectively.

Dissemination is carried 
out properly for strategic 
evaluations and ROM findings

59 
Strategic evaluation and ROM reports 
are easily accessible to all EuropeAid’s 
units and EU delegations. Strategic 
evaluation reports are also available 
to external stakeholders. They are 
published on the websites of Europe‑
Aid and the OECD, and are sent to the 
European Parliament, the Council and 
EU Member States. The reports and 
lessons learned are also disseminated 
in a variety of ways (e.g. meetings, 
workshops and seminars) adapted to 
the target audience (e.g. Commission 
departments, EEAS, EU delegations, 
European institutions, external part‑
ners, and networks of experts).

60 
EuropeAid presents a synthesis of 
strategic evaluation results and the 
key lessons learned, as well as the ag‑
gregated ROM performance data in its 
AAR50 and AR51. It also contributes to 
the preparation of the Commission’s 
Article 318 evaluation report which 
provides summaries of strategic evalu‑
ation findings52.

23

53  AAR 2013, Annex 9: 

Performance information 
included in evaluations.

54  Special Report No 1/2011, 

paragraph 60.

64 
EuropeAid has stated its intention to 
develop a new IT tool to manage, store 
and analyse ROM and programme 
evaluations, which would serve as 
a central storage point for all reports 
and allow users to perform advanced 
searches and analysis. At the time 
of the Court’s audit, the decision to 
develop the part of the application 
for programme evaluations had not 
yet been made, even though report ac‑
cessibility was a more crucial issue for 
programme evaluations.

Observations

Dissemination of programme 
evaluations is inadequate

61 
Programme evaluation reports are 
generally shared and discussed by 
programme managers with directly 
interested stakeholders, i.e. mainly Eu‑
ropeAid’s geographical and thematic 
units and the implementing partner. 
They are rarely shared with beneficiar‑
ies and other donors.

62 
Delegations are required to report 
on lessons learned from programme 
evaluations in their half‑yearly EAMRs 
to EuropeAid. EAMRs reviewed by the 
Court contained no or little informa‑
tion in this respect. For programme 
evaluations managed by EuropeAid’s 
operational units, there was no similar 
reporting requirement until 2012. In 
EuropeAid’s 2013 AAR53 a summary of 
15 of these programme evaluations is 
provided; however, what this repre‑
sents in terms of total evaluations 
completed in 2013 is not indicated.

63 
As the Court has pointed out54, pro‑
gramme evaluation reports cannot be 
systematically used because there is 
no central database for holding such 
evaluations. This reduces significantly 
EuropeAid’s capacity for analysing and 
disseminating this information inter‑
nally and to external stakeholders and 
for providing a basis for accountability.

Conclusions and 
recommendations

24

65 
The Court concludes that EuropeAid 
evaluation and ROM systems are not 
sufficiently reliable.

66 
Overall, the evaluation and ROM 
functions are well organised, with an 
appropriate combination of central‑
ised and decentralised responsibilities 
which aim to promote a culture of 
evaluation and give evaluation a prom‑
inent role in the aid management 
process. However, there is inadequate 
overall supervision of programme 
evaluations by senior management. In 
the absence of proper management 
information on staff and financial 
resources used for programme evalu‑
ations, EuropeAid cannot ensure that 
they are appropriate and are efficiently 
allocated to the various evaluation 
activities.

67 
The evaluation and ROM systems do 
not sufficiently ensure the delivery of 
relevant and robust evaluations and 
ROM findings. Evaluation plans, drawn 
up on the basis of insufficiently clear 
criteria, do not guarantee that prior‑
ity is given to those evaluations which 
are most useful for decision‑making. 
In addition, the absence of monitoring 
prevents EuropeAid from identifying 
and addressing the causes of frequent 
deviations from these plans. Procure‑
ment procedures enable external 
experts with the required skills and 
knowledge to be selected. Quality 
control procedures for ROM and pro‑
gramme evaluations are not applied 
consistently within EuropeAid.

68 
Maximum use is not made of evalu‑
ations and ROM findings. Proper 
follow‑up mechanisms are not in place 
to ensure that evaluation and ROM 
findings are appropriately used. Strate‑
gic evaluations and ROM findings are 
widely disseminated through a variety 
of communication tools to a wide 
range of internal and external stake‑
holders. However, the lack of access 
to programme evaluations and the 
absence of search tools are obstacles 
to the proper sharing of knowledge.

69 
The evaluation and ROM systems do 
not provide adequate information on 
results achieved due to insufficiently 
well‑defined objectives and indica‑
tors, the limited proportion of ex post 
evaluations and ROMs and inherent 
limitations in the evaluation method‑
ology for budget support. As a result, 
EuropeAid’s capacity to account for 
its activities is limited and its report‑
ing gives few indications of the actual 
results achieved.

70 
Recently, the Commission and Euro‑
peAid launched initiatives to improve 
their capacity to assess and report on 
policy achievements. These initiatives 
were at too early a stage for the Court 
to assess whether they were effective‑
ly addressing the issues raised in the 
present report.

Conclusions and recommendations

25

Recommendation 1 
Efficient use of resources 
for evaluation and ROM 
systems

EuropeAid should maintain adequate 
management information and perform 
needs assessments on a regular basis 
in order to ensure an informed alloca‑
tion of financial and staff resources 
between programme evaluations and 
ROM.

Recommendation 2 
Appropriate prioritisation of 
programme evaluations and 
monitoring of evaluation 
activities

In order to ensure that evaluations 
carried out reflect the priorities of the 
organisation, EuropeAid should:

—  define clear selection criteria to 

prioritise programme evaluations 
and document how they were 
applied in establishing the evalua‑
tion plans, taking into account the 
complementarity with ROM;

—  significantly strengthen its system 
to monitor and report on the im‑
plementation of evaluation plans, 
including an analysis of the rea‑
sons for delays and the description 
of measures adopted to address 
them;

—  strengthen the overall supervision 
of programme evaluation activities 
by EuropeAid.

Recommendation 3 
Quality control

In order to ensure the quality of 
programme evaluations and ROMs, 
EuropeAid should:

— 

insist that operational units and 
delegations apply the quality 
control requirements, including for 
programme evaluations the use of 
a reference group and documenta‑
tion of quality controls performed;

—  check, on a regular basis, the ap‑
plication of these controls.

Recommendation 4 
Demonstrating results 
achieved

In order to enhance the evaluation 
system’s capability to provide ade‑
quate information on results achieved, 
EuropeAid should:

—  apply more rigorously the regula‑
tory provisions requiring the use 
of SMART objectives and verifiable 
indicators;

—  modify the monitoring system 
so that it continues to provide 
data on programmes until at least 
3 years after their completion; and

— 

increase significantly the pro‑
portion of ex post programme 
evaluations.

Conclusions and recommendations

26

Recommendation 5 
Follow-up and 
dissemination of evaluation 
and ROM findings

In order to ensure the maximum use of 
evaluation and ROM findings, Europe‑
Aid should:

—  extend the follow‑up period for 

strategic evaluations;

—  develop a central database for 

programme evaluations to store 
evaluations reports, action plans 
and their follow‑up, with search 
functionalities which permit 
analysis.

This Report was adopted by Chamber III, headed by Mr Karel PINXTEN, Member 
of the Court of Auditors, in Luxembourg at its meeting of 21 October 2014.

For the Court of Auditors

Vítor Manuel da SILVA CALDEIRA 
President

 
 
 
Annexes

27

ROM grading system

I
x
e
n
n
A

ROM assesses a programme performance in respect of the five evaluation criteria (relevance, efficiency, effec‑
tiveness, impact prospects and potential sustainability) according to the following grading system:

Grading

Numerical

Qualitative

Explanation for the grade

A

B

C

D

4

3

2

1

Very good

Good

Problems

The situation is considered highly satisfactory, largely above average and potentially 
a reference for good practice. Recommendations focus on the need to adopt these 
good practices in other operations.

The situation is considered satisfactory, but there is room for improvements. Recom-
mendations are useful, but not vital for the operation.

There are issues which need to be addressed; otherwise the global performance of 
the operation may be negatively affected. Necessary improvements however do not 
require major revisions of the operations’ strategy.

Serious deficiencies

There are deficiencies which are so serious that, if not addressed, they can lead to 
failure of the operation. Major adjustments and revision of the strategy are necessary.

Each programme is then categorised, according to the number of A, B, C and Ds it has received. There are four 
categories in which a programme can be classified. ‘I’ indicates very good performance; ‘II’ indicates good per‑
formance; ‘III’ indicates that a programme is performing with problems; and ‘IV’ indicates that a programme is 
not performing, or that it is experiencing major difficulties.

EuropeAid then determines the percentage of programmes in the ROM portfolio falling under each category. 
This is done for the whole portfolio, per region and per sector within a region. This is the basis for its reporting 
on the portfolio performance.

 
Annexes

28

I List of programming documents reviewed
I
x
e
n
n
A

Thematic

 ο

 ο

 ο

Environment and Natural Resources Thematic Programme (ENRTP): strategy paper and multiannual indicative 
programme (MIP) 2011–13

Food security thematic programme, strategy paper and MIP 2011–13

Investing in people: Mid‑term Review of strategy paper for Thematic Programme 2007–13 and MIP 2011–13

Regional

 ο

 ο

 ο

 ο

Central Asia Regional Strategy Paper 2007–13

Latin America Regional Indicative Programme 2007–13

Asia MIP 2007–10

Intra ACP cooperation 10th EDF, MIP 2008–13

Country

 ο

 ο

 ο

 ο

 ο

 ο

 ο

 ο

Country Strategic Paper (CSP) Laos 2007–13

CSP Bangladesh 2007–13

CSP India 2007–13

CSP/National Indicative Programme (NIP) Benin 2008–13

CSP/NIP Chad 2008–13

CSP/NIP Zambia 2008–13

CSP/NIP Uruguay 2007–13

CSP/NIP Bolivia 2007–13

 
Annexes

I List of financing agreements reviewed
I
I
x
e
n
n
A

Decision number

Country

Title

Africa

Madagascar

FED/2012/023-503 (EC)

FED/2013/023-513 (EG)

FED/2011/023-531 (CA)

Amélioration de la Sécurité Alimentaire et Augmentation des Revenus 
Agricoles (ASARA)

Small and Medium Agribusiness Development Fund — A Public–Private 
Partnership.

Capacity development and governance support for scaling up renewable en-
ergy services in rural and peri-urban areas of Ethiopia — CfP 129364 Prop 353

FED/2012/023-536 (EC)

Support to Democratisation

Uganda

Ethiopia

Somalia

Papua New Guinea

FED/2012/023-538 (EG)

Non State Actors (NSAs) Support Programme in Papua New Guinea

Nigeria

Kenya

Liberia

Latin America

Honduras

Costa Rica

FED/2013/023-551 (EG)

Energising Access to Sustainable Energy in Nigeria (EASE)

FED/2012/023-566 (EC)

Standards for Market Access Programme (SMAP)

FED/2012/023-568 (EC)

Support to trade facilitation in the customs sector

8 Financing Agreements 

TOTAL

DCI-ALA/2013/023-720 (EC)

Promoting fair and accessible justice in Honduras (EuroJusticia)

DCI-ALA/2010/021-501 (EC)

Comprehensive strategy to reduce school drop-out rates in public secondary 
education

Dominican Republic

BAN/2012/024-100 (EC)

Banana accompanying measure (BAM) Dominican Republic

Uruguay

DCI-ALA/2011/022-498 (EC)

Support for the reform of the Uruguayan penal justice and penitentiary 
systems for the improvement of the quality of life and social and labour 
reintegration of prison inmates

4 Financing Agreements 

TOTAL

Neighbourhood

Egypt

Morocco

Tunisia

West Bank and Gaza 
Strip

Asia and Pacific

Afghanistan

Tonga

Vietnam

ENPI/2012/024-080 (EC)

Emergency Employment Investment Project

ENPI/2012/024-196 (EC)

Programme d’appui à l’Initiative Nationale de Développement Humain (INDH) 
— phase II

ENPI/2011/022-772 (EC)

Programme de Modernisation des services

ENPI/2010/022-476 (EC)

PEGASE: Private Sector Development

4 Financing Agreements 

TOTAL

DCI-ASIE/2013/024-844 (EG)

Support to credible and transparent elections

FED/2013/024-766 (EG)

Technical Cooperation Facility, Good Governance and NSA programme II

DCI-ASIE/2013/024-370 (EG)

EU-HSPSP 2 — EU Health Sector Policy Support Programme phase 2: Towards 
Equity and Quality of Health Services in Vietnam

Bangladesh

DCI-ASIE/2012/022-446 (EC)

Poverty Reduction Through Inclusive and Sustainable Markets (PRISM)

4 Financing Agreements 

20 Financing Agreements 

TOTAL

TOTAL

29

(euro)

EU 
contribution

40 000 000

15 000 000

1 176 000

14 000 000

4 538 000

27 000 000

12 100 000

3 000 000

116 814 000

27 500 000

8 500 000

16 340 000

5 000 000

57 340 000

70 000 000

25 000 000

20 000 000

11 000 000

126 000 000

15 000 000

472 000

114 000 000

30 000 000

159 472 000

459 626 000

 
Annexes

30

(euro)

List of evaluations reviewed

V

I
x
e
n
n
A

2011

2012

2013

2013

2012

2013

2011

2012

2010

Programme

Final

Programme 
but similar to 
Strategic

Programme 
but similar to 
Strategic

Final

Final

Programme

Final

Programme

Final

Programme

Final

Year1

Programme 
or Strategic 
Evaluations

Evaluation 
type

2012

Strategic

Country

Programme/theme/country evaluated

Financing 
instrument

EU 
contribution

Evaluation 
cost

European Union’s cooperation with Nepal 
— Country Level Evaluation (covered period 
2002–10)

Mainly DCI geographic 
and thematic and EIDHR

58 000 000

201 711

Strategic

Thematic

European Commission support in the educa-
tion sector (covered period 2000–07)

DCI geographic and 
thematic, EDF, ENPI

1 900 000 000

457 265

Food Facility — AU-IBAR projects (LEISOM 
and VACNADA)

Final evaluation of the whole Food Facility 
(2009–12)

Food facility

2 507 860

131 870

Food facility

1 000 000 000

399 720

Three Year Action Programme for the African 
Peace Facility for the period 2011–13.

Mainly EDF

Programme

Mid term

‘Invest in MED’

Programme

ex post

‘Euromed Migration II’

ENPI South

ENPI South

Appui à l’Entretien Périodique du Réseau 
Routier Classé 

Programme d’appui à la décentralisation au 
Bénin

EDF

EDF

549 345 726

478 022

9 000 000

4 994 000

119 778

148 580

36 709 425

72 848

13 785 026

47 120

Erasmus Mundus, External Cooperation 
Window for India

EC assisted State Partnership Programme 
to the states of Chhattisgarh and Rajasthan 
(budget support)2

20 projects selected among 100 on EU 
Funded Environment Projects in India 

DCI geographic

28 600 000

147 745

DCI geographic

158 590 819

1 934 700

DCI geographic

340 000 000

118 008

2014

Programme 

Final

2011

Programme

Final

1  Year when the evaluation was issued.
2  The contract entails more than just an evaluation.

 
Annexes

31

V
x
e
n
n
A

List of ROMs reviewed

Programme Title

Food Facility — AU-IBAR projects (LEISOM 
and VACNADA)

FOOD FACILITY

Country/region/ 
sector

Fighting hunger

Fighting hunger

Public Administration Capacity Building 
Facility Libya

Libya

PROJECT ‘Euromed Migration II’

Neighbourhood South

Appui à l’Entretien Périodique du Réseau 
Routier Classé 

Programme d’appui à la décentralisation 
au Bénin

Programme Société Civile et Culture

Sustainable Textiles for Sustainable 
Development

European Business and Technology Centre 
in India (EBTC)

Benin

Benin

Benin

India

India

EU contribution

Report type

Report date

(euro)

2 507 860

1 000 000 000

4 500 000

4 994 000

36 709 425

13 785 026

14 400 000

1 672 945

6 586 578

Ongoing

Ongoing

Ongoing

Ongoing

Ongoing

Ongoing

Ongoing

Ongoing

Ongoing

June 2011

May 2011

October 2013

October 2010

November 2008

February 2012

May 2013

July 2012

July 2012

 
Annexes

32

I List of delegations which replied to the survey
V
x
e
n
n
A

Country/Delegation

Africa Caribbean Pacific

Latin America

Bolivia

Chile

El Salvador 

Honduras 

Nicaragua 

Paraguay 

Uruguay

Neighbourhood

Algeria 

Armenia 

Azerbaijan 

Belarus

Georgia 

Israel 

Jordan 

Lebanon

Morocco 

Tunisia 

Ukraine 

Angola 

Botswana

Burkina Faso

Burundi

Cape Verde 

Congo (Brazzaville) 

Djibouti 

Ethiopia 

Madagascar 

Nigeria 

Rwanda 

Togo

Dominican Republic 

Guyana 

Solomon Islands (Del. Papua New Guinea)

Asia

Bangladesh

Cambodia

India 

Laos, Thailand

Nepal 

Pakistan 

Philippines 

Vietnam 

 
Reply of the  
Commission

33

Executive summary

II
The Commission considers that the systems for 
strategic evaluations are overall reliable even if they 
could be improved. For programme evaluations 
and ROM, systems exist but in some areas are not 
functioning properly.

III
The systems exist for supervision of programme 
evaluations, and steps are being taken to make 
them more effective. 

The Commission considers that financial allocations 
are well managed for strategic evaluations and ROM 
and that improvements are possible for programme 
evaluations.

The human resources management for evaluation 
and ROM should be considered in the framework of 
the overall human resources management for the 
external assistance managed by EuropeAid. 

IV
The Commission’s evaluation findings are based on 
internationally recognised methodology agreed in 
the OECD. EuropeAid’s evaluation plans are drawn 
up to respond to the aims of learning and accounta‑
bility, are implemented in accordance with common 
principles of evaluation and are monitored through 
regular internal reports.

Guidance is going to be provided to staff on 
drawing up plans for programme evaluations and 
monitoring activities, ensuring complementarity 
between them. The selection of projects to be 
subject of a ROM will, as from January 2015, take 
into account better, whether or not an evaluation is 
foreseen.

A system for monitoring the implementation of 
programme evaluation plans already exists in the 
EAMR, and progress in strategic evaluations is 
reported to Commissioners.

The monitoring system for programme evaluations 
will be improved and more emphasis will be put on 
quality control issues for the programme evalua‑
tions. Improved guidance and assistance will also 
be offered to the project managers through the 
information management system for programme 
evaluations which is under development.

With respect to ROM, the Commission has initiated 
reforms that should substantially improve the qual‑
ity of the system, including in particular external 
quality assurance.

V
Mechanisms for follow‑up and dissemination are 
already in place. For ROM, the existing mechanism 
is being improved. Follow‑up to strategic evalua‑
tions is formalised and published. 

For programme evaluations, the Commission recog‑
nises that improvements are possible with respect 
to follow‑up, oversight and monitoring, functionali‑
ties that will be covered by the information man‑
agement system under construction, providing 
moreover appropriate access to programme evalua‑
tions, and ensuring better dissemination.

VI
The smart objectives and verifiable indicators are 
defined at the level of multiannual programming 
and project documents.

The choice on whether to carry out an ex post evalu‑
ation is linked to criteria of efficiency and relevance. 
The number of individual ex post evaluations as 
such has no relevance.

Reply of the Commission 

34

The Commission discontinued ex post ROM under 
the revised ROM system as it is not a monitoring 
tool and as it is moreover not considered to be an 
appropriate tool to assess impact.

The Commission considers that there are no inher‑
ent limitations in the evaluation methodology for 
budget support.

Introduction

03
The latest (fourth) Article 318 Report was adopted 
on 26 June 2014. It builds on the experience gath ‑
ered with the previous editions of the report and 
takes due account of the recommendations made 
by the Court of Auditors and the European Parlia‑
ment, including those made in the EP resolution of 
26 February 2014. 

09
The main system used by Europe‑Aid to moni‑
tor programme implementation on the spot is 
the internal monitoring by the Commission’s own 
services and by EU delegations in accordance with 
the 2007 guidelines on internal monitoring and the 
2012 budget support guidelines. ROM has therefore 
been an additional system of programme perfor‑
mance monitoring implemented by external con‑
sultants with respect to programmes implemented 
through the project modality. 

Observations

The human resources management for evaluation 
and ROM should be considered in the framework 
of the overall human resources management for 
the external assistance managed by EuropeAid.

17
The delegations and operational units already 
report to EuropeAid Management via the annual 
evaluation plan in the Annual Management Plan, 
and report systematically to headquarters on the 
implementation of their evaluation plan in their 
EAMR. 

Instructions and guidance are under preparation 
to staff on the preparation and implementation of 
their future monitoring and evaluation plans and 
to the follow‑up to be given to the conclusions 
and recommendations of evaluations. Moreover an 
information management system is under devel‑
opment that will facilitate the management and 
monitoring of programme evaluations. The network 
of Evaluation Correspondents in delegations and 
operational units, set up in 2013, will provide added 
feedback.

The human resources management for evaluation 
and ROM should be considered in the framework of 
the overall human resources management for the 
external assistance managed by EuropeAid.

19
The information management system under con‑
struction will provide more precise information on 
the number of programme evaluations and their 
cost.

20
Monitoring, reporting, mid‑term evaluation and, 
more generally, project management activities are 
so closely interlinked that any division of resource 
allocation between them can only be an estimate. 
This is even more so with respect to the new ROM 
which is to be a tool for support to overall monitor‑
ing of programmes.

21
The human resources management for evaluation 
and ROM should be considered in the framework of 
the overall human resources management for the 
external assistance managed by EuropeAid.

22
Avoiding ROM when an evaluation is to take place 
was already a principle before the ongoing reform 
of monitoring and evaluation systems (as men‑
tioned also by the Court). This reform also includes 
the introduction of criteria for the selection and 
planning of programme evaluations. Furthermore, 
a new requirement for delegations and opera‑
tional units to draw up an annual monitoring and 

Reply of the Commission 

35

evaluation plan will be introduced, ensuring com‑
plementarity between ROM reviews and mid‑term 
programme evaluations.

The Commission considers that EuropeAid’s evalua‑
tion and ROM systems provide relevant and robust 
findings, even if improvements are needed.

26
Instructions and guidance to staff on the selection 
criteria for programme evaluations and the plan‑
ning of evaluations within the framework of the 
preparation of monitoring and evaluation plans will 
be part of the overall instructions and guidance on 
the reforms of monitoring, reporting and evalua‑
tion. This guidance will be issued before the end 
of 2014. Selection criteria are to be reflected in the 
financing agreements.

26 (a)
Mid‑term evaluations are made compulsory on 
a case‑by‑case basis at formulation phase taking 
into account the programme specificity.

26 (b)
Ex post evaluations are made compulsory on a case‑
by‑case basis at formulation phase taking into 
account the programme specificity.

Evaluations are delivered as planned in a majority 
of cases.

27
Systems are in place but the Commission acknowl‑
edges the need to improve them.

28
In addition to staff constraints, fewer strategic 
evaluations than planned were completed because 
of the political situation in various countries, the 
increasing complexity of evaluations (particularly 
evaluations carried out jointly with partners and 
other donors), and also to changes in the contrac‑
tual framework for these evaluations. 

29
The Commission reminds that the implementation 
rate is above 80 %, which is the EuropeAid's bench‑
mark (KPI 19 EAMR).

The EAMR will have to provide improved documen‑
tation of the non‑compliance of the planning of 
the programme evaluations that will be established 
by Delegations and operational HQ services in the 
upcoming Monitoring and evaluation plans.

31
The revised new ROM system as adopted in May 
2014 as part of a wider reform of monitoring and 
reporting has the primary objective of improving 
the internal monitoring and reporting on perfor‑
mance and results by delegations and operational 
units and of reducing a too heavy reliance on 
external ROM, in line with the Court's special report 
No 1/2011 on devolution. The main objective of 
the new ROM system is to provide support to the 
internal monitoring and reporting. The selection 
methodology reflects these priorities:

—  For internal monitoring purposes, the selection 
methodology for ROM review focuses essen‑
tially on projects and programmes flagged by 
delegations and operational units as (1) facing 
problems, (2) for which sector expertise is lack‑
ing and (3) that could not be visited.

—  For reporting on results purposes, the system 
is to support end of projects results reporting, 
covering all projects and programmes above 
EUR 750 000 ending in a given year and to assist 
delegations and operational units to report on 
results, including for the aggregated results on 
the basis of the Commission’s future EU Devel‑
opment and Cooperation Results framework.

The Commission therefore considers that by includ‑
ing a risk‑based selection for ongoing programmes, 
while ending the use of ROM as an indicator for the 
overall performance measurement of its portfolio, 
the new system will enhance the usefulness of ROM, 
in accordance with its main objective. While ROM 
is not designed for the purpose of measuring the 
overall performance of the Commission’s aid portfo‑
lio, the strengthening of the internal monitoring 

Reply of the Commission 

36

and reporting on performance and results and the 
planned reporting on aggregated results on the 
basis of the Commission’s future EU Development 
and Cooperation Results framework are to be part 
of the tools that should allow the Commission 
to better measure the overall performance of its 
portfolio.

33
Under the new ROM contracts to be signed in 2014, 
all the reviews and other support missions are to 
be implemented by senior experts who should 
moreover be sector experts. Besides, a significant 
number of reviews should indeed be carried out by 
key experts on a full time assignment whereas for 
the other reviews the experts will be hired on an ad 
hoc basis.

36
EuropeAid has emphasised in its recent Evalua‑
tion Policy the importance of adhering to existing 
guidance on setting up reference groups. (See for 
existing guidelines: http://ec.europa.eu/europeaid/
evaluation/methodology/guidelines/gba_det_
en.htm#01_03 and http://ec.europa.eu/europeaid/
evaluation/methodology/examples/131011‑tor‑pro‑
ject‑evaluations‑guidance_en.pdf), 

37
The introduction in 2013 of the Evaluation Corre‑
spondents’ network aims to strengthen significantly 
the evaluation function across the system, including 
in Delegations.

More emphasis will be put on quality control issues 
for the programme evaluations.

38
Several measures to be taken under the revised 
ROM system should solve the deficiencies charac‑
terising the present situation: the strengthening of 
the quality of ROM reviews, the improved condi‑
tions for appropriate internal quality control by the 
ROM contractors, the setting up of external quality 
assurance, the importance to be given to defining 
the follow‑up, and improved overall monitoring of 
implementation of the revised ROM system. The 
encoding of the follow‑up planned to take place in 
the new information management module for ROM 
is another measure that should contribute to the 
same. 

The Commission considers that the evaluation and 
ROM systems provide adequate information on 
results achieved, even if improvements are needed.

For the new programming period 2014–20, the Com‑
mission has stepped up its efforts to have smart objec‑
tives and verifiable indicators. The new programming 
documents have been reviewed in that sense and the 
operationalisation of those efforts at project level has 
been initiated.

40
Objectives and indicators for regional and world‑
wide thematic programmes have to be general in 
order to capture a variety of situations. (see reply to 
Box 1).

Box 1
Investing in people is a worldwide thematic pro‑
gramme covering all developing countries. This 
implies that the objectives are in many cases 
rather high level precisely to cover many different 
situations.

The objective quoted for the 2007–13 Bolivia 
programming document represents rather an 
overall (or general) objective than a set of specific 
objectives.

Reply of the Commission 

37

41
In programmes covering several countries and 
regions, the objectives of the programme give the 
key priorities; the specific objectives for each sub‑
programme are more precise and indicators should 
be measurable.

Box 2 
The Commission wants to highlight that these 
examples refer to general objectives.

Concerning the Euromed migration programme, 
the objectives are only partially quoted by the 
Court. The full objectives can be found at the fol‑
lowing address: http://www.enpi‑info.eu/mainmed.
php?id_type=10&id=9

42
The purpose of the study to which the Court refers 
was to review the evaluations related to the legal 
instruments for the last programming period (2007–
13). The findings of the study have been taken into 
account to prepare the instruments covering the 
MFF 2014–20.

Strategic evaluations focus on the chain of 
results achieved and not on implementation of 
programmes.

Common Commission reply to 
paragraphs 43 to 45
The decision to undertake an ex post evaluation 
of a programme is a question of efficiency and 
relevance. The decision should not be taken in 
isolation but with reference to the financial weight 
and strategic importance of the programme to be 
evaluated. 

46
Under the revised ROM system, the Commission 
discontinued ex post ROM as it is not a monitoring 
tool and is not considered to be an appropriate tool 
to assess impact. 

47
The approach of the EIB referred to by the Court 
is specific and is to be seen in connection with 
the nature of the loan‑based operations managed 
by the EIB. It cannot be compared to the case of 
a grant donor like the EU for the grant based fund‑
ing of programmes.

The ongoing set‑up of the EU Development and 
Cooperation Results Framework through which the 
Commission aims to implement a first reporting in 
2015 does not proceed on such an approach.

Methodology to evaluate budget support has 
proved its strong results and has been recognised at 
international level.

48
The document setting out the methodological 
approach to evaluating budget support developed 
under the auspices of the OECD‑DAC, provides 
a robust approach addressing the complexity of 
evaluating budget support.

49
Robustness of the approach is recognised by the 
international evaluation community, and it has 
been published on the OECD website.

Consequently, the Commission does not see any 
reason to mention any limitations in its methodol‑
ogy. So far, seven evaluations of budget support 
have been finalised which have been able to assess 
the results of budget support and have made rec‑
ommendations for improvement.

Evaluations are a tool for assessment of the whole 
chain of results, but as regards outputs and direct 
outcomes internal monitoring provides primary 
information and is used to ensure reporting on 
results across the portfolio. 

50
The Commission considers that ROM is not the 
appropriate tool to provide information on sus‑
tainability and impact and that ex post evaluations 
should be only decided on a case‑by‑case basis.

Reply of the Commission 

38

52
When preparing the latest (fourth) edition of 
the Article 318 Evaluation Report, adopted on 
26 June 2014, the Commission has duly taken into 
account the observations made by the Court of 
Auditors in its 2012 annual report. 

The Commission also refers to its replies to the 
Court’s observations under Chapter 10 of the ECA 
2012 Annual Report where it provides its position 
with regard to the performance information in the 
third and future editions of the Art. 318 evaluation 
report.

Follow-up mechanisms are in place, although they 
could be improved.

54
Given the complexity of strategic evaluations, it is 
difficult to have clear cut commitments linked to 
high level recommendations. In 2013, the Evalua‑
tion unit carried out a study on uptake of strategic 
evaluations results. On that basis, the management 
response system is being reviewed. In any case, the 
responsibility lies clearly with the geographic or 
thematic services responsible for the implementa‑
tion of the programmes or policies. 

55
The review of the management response system 
will take into account not only this issue of the dura‑
tion of the follow‑up period but also other issues 
like brokering the evaluation results and involve‑
ment of management.

56
The Evaluation unit does know which recommen‑
dation has been accepted. The completed ‘fiches 
contradictoires’ indicate clearly whether a given 
recommendation is accepted, partially accepted 
or not accepted. On the basis of the study referred 
to by the Court, the Commission has adopted an 
action plan to improve the uptake of strategic 
evaluation results. 

The Commission will take additional measures to 
improve the documentation of follow‑up for pro‑
gramme evaluations and ROM.

57
Follow‑up to programme evaluations is part of the 
the lessons learnt section of new project propos‑
als (identification and formulation stages of new 
projects) when deemed relevant.

The guidance note accompanying the updated 
terms of reference template for programme evalu‑
ations issued in 2013 emphasised the need to 
document the management response. For ROM, the 
revised system is to include appropriate measures 
to ensure the definition of an appropriate follow‑
up plan to a ROM review and related monitoring of 
such follow‑up.

For programme evaluations, the point will be taken 
into account in the upcoming instruction guidance 
as well as in the information management system 
under construction.

58
For ROM, as stated in the reply to §59, the revised 
system is to include appropriate measures to ensure 
the definition of an appropriate follow‑up plan 
to a ROM review and related monitoring of such 
follow‑up.

The Commission is in the process of develop-
ing an information management system needed 
for improving dissemination of programme 
evaluations.

60
The aggregation of ROM performance data in its 
previous form in AAR and AR will be discontinued. 

Reply of the Commission 

39

The Commission is in the process of developing the 
related information management system needed 
for improving such dissemination.

61
Dissemination of programme evaluations will be 
improved:

—  Firstly, by improved access to such evaluations 
through the setting up of an information man‑
agement system;

—  Secondly, on that base by improved analysis of 

such evaluations by thematic units; and

—  Finally, by better contacts between Delegations 

and in‑country stakeholders.

The Evaluation Correspondents’ Network can play 
an important part in this.

62
The information management system that is to con‑
tain the database for programme evaluations is also 
to contain the lessons learned.

63
The Commission is in the process of developing 
such an information management system needed 
for improving such dissemination.

Conclusions and recommendations

65
The Commission considers that the systems for 
strategic evaluations are overall reliable even if they 
could be improved. For programme evaluations 
and ROM, systems exist but in some areas are not 
functioning properly.

66
The systems exist for supervision and use of pro‑
gramme evaluations, and steps are being taken to 
make them more effective. 

The Commission considers that financial allocations 
are well managed for strategic evaluations and ROM 
and that improvements are possible for programme 
evaluations.

The human resources management for evaluation 
and ROM should be considered in the framework of 
the overall human resources management for the 
external assistance managed by EuropeAid. 

67
The Commission’s evaluation findings are based on 
internationally recognised methodology agreed in 
the OECD. Guidance is going to be provided to staff 
on drawing up plans for programme evaluations 
and monitoring activities, ensuring complementa‑
rity between them. 

64
The IT tool covering both ROM and programme 
evaluations, was already under construction at the 
time of the audit.

A system for monitoring the implementation of 
programme evaluation plans already exists in the 
EAMR, and progress in strategic evaluations is 
reported to Commissioners.

This development has, in 2014, undergone a delay 
as a result of the overall revision of the direction to 
take with respect to EuropeAid’s new operational 
information management system that is to be 
developed. This does however not put into ques‑
tion the need to develop such a system.

With respect to ROM, the Commission has initiated 
reforms that should substantially improve the qual‑
ity of the system, including in particular external 
quality assurance.

Reply of the Commission 

40

Recommendation 1
The Commission accepts the recommendation. 

The human resources management for evaluation 
and ROM should be considered in the framework of 
the overall human resources management for the 
external assistance managed by EuropeAid.

Recommendation 2
—  The Commission accepts this recommendation. 
Guidance is to be provided to staff on drawing 
up plans for programme evaluations and moni‑
toring activities, ensuring complementarity 
between them. The selection of projects to be 
subject of a ROM will, as from January 2015, bet‑
ter take into account whether or not an evalua‑
tion is foreseen. 

—  The Commission accepts this recommendation. 
For programme evaluation plans, it will be done 
on the basis of the EAMR. 

—  The Commission accepts this recommendation.

Recommendation 3
—  The Commission accepts this recommendation. 
More emphasis will be put on quality control is‑
sues for the programme evaluations. Improved 
guidance and assistance will also be offered to 
the project managers through the information 
management system for programme evalua‑
tions which is under development.

—  The Commission accepts this recommendation. 
The monitoring system for programme evalua‑
tions will be improved.

68
Mechanisms for follow‑up and dissemination are 
already in place. For ROM, the existing mechanism 
is being improved. Follow‑up to strategic evalua‑
tions is formalised and published. 

For programme evaluations, the Commission recog‑
nises that improvements are possible with respect 
to follow‑up, oversight and monitoring, functionali‑
ties that will be covered by the information man‑
agement system under construction, providing 
moreover appropriate access to programme evalua‑
tions, and ensuring better dissemination.

69
The smart objectives and verifiable indicators are 
defined at the level of multiannual programming 
and project documents.

The choice on whether to carry out an ex post evalu‑
ation is linked to criteria of efficiency and relevance. 
The number of individual ex post evaluations as 
such has no relevance.

The Commission discontinued ex post ROM under 
the revised ROM system as it is not a monitoring 
tool and as it is moreover not considered to be an 
appropriate tool to assess impact.

The Commission considers that there are no inher‑
ent limitations in the evaluation methodology for 
budget support.

70
The revised ROM reform adopted by EuropeAid in 
May 2014 aims not only to clarify the distinctions 
between ROM and evaluation but also to make the 
system more efficient; this will be made operational 
in the detailed instructions to be sent to staff by the 
end of 2014. Following publication of the evalua‑
tion policy document ‘Evaluation matters’, reflect‑
ing a major process of consultation and discussion 
between October 2013 and March 2014, the guid‑
ance to staff on the principles, responsibilities and 
procedures for evaluations will be updated and 
reinforced. 

Reply of the Commission 

41

Recommendation 4
—  The Commission accepts this recommendation 
and promotes the use of SMART objectives and 
verifiable indicators.

—  The Commission does not accept this recom‑
mendation as the cost benefit is not proven.

—  The Commission accepts this recommendation, 
if further analysis shows an increase in ex post 
evaluations is efficient and useful. 

Recommendation 5
—  The Commission partially accepts this recom‑

mendation. Extension of the follow‑up period is 
one of the options being considered in a review 
of the follow‑up system.

—  The Commission accepts this recommendation. 
Such a database is already under construction. 

HOW TO OBTAIN EU PUBLICATIONS

Free publications:

• 

• 

 one copy: 
via EU Bookshop (http://bookshop.europa.eu);

 more than one copy or posters/maps: 
from the European Union’s representations (http://ec.europa.eu/represent_en.htm);  
from the delegations in non‑EU countries (http://eeas.europa.eu/delegations/index_en.htm);  
by contacting the Europe Direct service (http://europa.eu/europedirect/index_en.htm) or 
calling 00 800 6 7 8 9 10 11 (freephone number from anywhere in the EU) (*).

(*) 

The information given is free, as are most calls (though some operators, phone boxes or hotels may charge you).

Priced publications:

• 

via EU Bookshop (http://bookshop.europa.eu).

Priced subscriptions:

• 

 via one of the sales agents of the Publications Office of the European Union 
(http://publications.europa.eu/others/agents/index_en.htm).nts/index_en.htm).

Q
J
‑
A
B
‑
1
4
‑
0
1
8
‑
E
N

‑

N

I

S
S
N
1
9
7
7
‑
5
6
7
9

Evaluation and results-oriented monitoring 
(ROM) are part of EuropeAid’s results 
accountability framework. The Court found that 
EuropeAid’s evaluation and ROM systems are not 
sufficiently reliable. Overall, evaluation and ROM 
functions are well organised. However, some 
weaknesses in the design and implementation of 
the systems reduce EuropeAid’s capacity to 
assess and report on EU’s policy achievements.

EUROPEAN
COURT
OF AUDITORS