Source: EURLEX
Language: en
Format: md

**EUROPEAN COURT OF AUDITORS**
**12, rue Alcide De Gasperi**
**1615 Luxembourg**
**LUXEMBOURG**

**Tel. +352 4398-1**

**E-mail: eca-info@eca.europa.eu**
**Internet: http://eca.europa.eu**

**Twitter: @EUAuditorsECA**
**YouTube: EUAuditorsECA**

**More information on the European Union is available on the Internet (http://europa.eu).**

**Luxembourg: Publications Office of the European Union, 2014**

**ISBN 978-92-872-1898-8**

**doi:10.2865/44910**

**© European Union, 2014**
**Reproduction is authorised provided the source is acknowledged.**

_Printed in Luxembourg_

### **EN 2014**

**Special Report**

##### **EuropeAid’s evaluation** **and results‑oriented** **monitoring systems**

(pursuant to Article 287(4), second subparagraph, TFEU)

# **NO 18**

## **Contents 02**

**Paragraph**

**Glossary and abbreviations**

**I** **–** **VII**
**Executive summary**

**1** **–** **10** **Introduction**

**2–5** **The Commission’s evaluation framework**

**6–10** **EuropeAid’s evaluation and ROM systems**

**11** **–** **13** **Audit scope and approach**

**14** **–** **64** **Observations**

**14–22** **Overall, the evaluation and ROM functions are well organised but EuropeAid could pay more**
**attention to the efficient use of resources**

**15** **–** **17** **There is a sound distribution of responsibilities but insufficient supervision of programme evaluations**

**18** **–** **22** **EuropeAid does not give enough consideration to the efficient use of resources**

**23–52** **EuropeAid’s evaluation and ROM systems do not sufficiently ensure relevant and robust findings**

**24** **–** **31** **There are some weaknesses in the selection of programme evaluations and timely delivery of evaluations**

**32** **–** **38** **The procedures to ensure the quality of programme evaluations and ROMs are implemented inconsistently**

**39** **–** **52** **The evaluation and ROM systems do not provide adequate information on results achieved**

**53–64** **EuropeAid’s evaluation and ROM systems do not ensure the maximum use of the findings made**

**54** **–** **58** **Proper follow‑up mechanisms are not in place**

**59** **–** **64** **Evaluation and ROM findings are well disseminated except for programme evaluations**

**65** **–** **70** **Conclusions and recommendations**

## **Contents 03**

**Annex I** **— ROM grading system**

**Annex II** **— List of programming documents reviewed**

**Annex III — List of financing agreements reviewed**

**Annex IV — List of evaluations reviewed**

**Annex V** **— List of ROMs reviewed**

**Annex VI — List of Delegations which replied to the survey**

**Reply of the Commission**

## **Glossary
 04**
#### **and abbreviations**

**Annual Report on the European Union’s Development and External Assistance Policies and their**
**Implementation (AR)** : This report is requested in the regulations establishing the financing instruments for the
EU’s external assistance and has to be submitted to the European Parliament, the Council, the European Economic
and Social Committee and to the Committee of the Regions. It is a report on the achievement of the objectives of
each financing instrument, is based on indicators and measurements of the results delivered and the efficiency of
the relevant instruments. It can be found at: http://ec.europa.eu/europeaid/annual‑report-2014-european‑unions‑
development‑and‑external‑assistance‑policies‑and‑their_en

**Annual Activity Report (AAR)** : This report presents the achievements reached and resources used by the
Directorate-General. It is the ‘mirror’ of the annual work programme. It is also a management report of the
Director‑General to the Commission on the performance of his duties as Authorising Officer by delegation, in the
management of the EU budget. This report is also communicated to the European Parliament and the Council. It can
be found at: http://ec.europa.eu/atwork/key‑documents/index_en.htm

**Article 318 evaluation report** : Article 318 of the Treaty on the Functioning of the European Union requires the
Commission to ‘submit to the European Parliament and to the Council an evaluative report on the Union’s finances
based on the results achieved’. The report can be found at: http://ec.europa.eu/smart‑regulation/evaluation/
documents_en.htm

**Budget support** : This method of aid delivery involves transferring funds from an external financing agency to the
partner government’s national treasury. The funds thus transferred are managed in accordance with the recipient’s
budgetary procedures. Budget support programmes usually include capacity development measures to help
partner countries strengthen their policymaking and management, as well as to promote dialogue on the design,
implementation and results of national and sectorial policies.

**External Assistance Management Report** : EU delegations’ main reporting document to EuropeAid. It has to be
made available to the European Parliament and the Council.

**European Investment Bank (EIB)** : The EIB is the EU’s bank. It is owned by and represents the interests of the EU
Member States. It works closely with other EU institutions to implement EU policy. It is the largest multilateral
borrower and lender by volume in the world. More than 90 % of its activity is focused on Europe, but it also
supports the EU’s external and development policies.

**European Development Funds (EDFs)** : The EDFs are the main instrument for providing EU aid for development
cooperation to African, Caribbean and Pacific (ACP) States and overseas countries and territories (OCTs). The
partnership agreement signed in Cotonou on 23 June 2000 for a period of 20 years is the current framework for
the EU’s relations with ACP States and OCTs. Its main focus is on reducing and eventually eradicating poverty.
The EDFs are funded by the EU Member States, governed by their own financial regulations and managed, by the
Commission, outside the EU general budget.

**European External Action Service (EEAS)** : The EEAS was launched in December 2010 under the Treaty of Lisbon.
Among its activities, it prepares country and regional allocations and EU cooperation strategies jointly with relevant
Commission departments, as part of the programming cycle for most external action instruments.

## **05**

**Glossary and abbreviations**

**EuropeAid** : Within the European Commission, the Directorate‑General for Development and Cooperation EuropeAid is responsible for:
— formulating EU development policy and defining sectoral policies in the field of external aid;
— drawing up the multiannual programming of the external aid instruments together with the European External

Action Service (EEAS);
— fostering coordination between the EU and the Member States on development cooperation and externally

representing the EU in this field.

**Evaluation findings** : For the purpose of this report, this term covers conclusions, recommendations and lessons
learned from an evaluation.

**EU delegations** : The EU is represented through 139 EU delegations and offices around the world which present,
explain and implement EU policies (e.g. the development cooperation policy), analyse and report on policies and
developments of the countries to which they are accredited, and conduct negotiations in accordance with a given
mandate.

**Internal Audit Capability** : is a unit of a Commission’s directorate‑general which reports directly to the
director‑general. Its task is to provide independent assurance on the effectiveness of the internal control system
with a view to improving the directorate‑general’s operations.

**Financing instruments** : These are regulations, usually issued by the European Parliament and the Council. They
provide the financing needed to support an EU policy. The EU’s development policy is supported by several
financing instruments.

**Financial perspectives** : A multiannual financial planning framework which sets limits on EU expenditure. It puts
the priorities set for the Union’s policies into financial terms and is at the same time an instrument of budgetary
discipline and planning. The current financial perspectives cover the period from 2014 to 2020; the previous one
covered the period from 2007 to 2013.

**OECD‑DAC** : Development Assistance Committee. A committee within the OECD (Organisation for Economic
Cooperation and Development) dealing specifically with development assistance.

**Result** : This term covers:
— outputs: that is produced or accomplished with the resources allocated to an intervention (e.g. grants

distributed to farmers, training courses delivered to unemployed people, road built);
— outcomes: the immediate changes resulting from an intervention (e.g. improved accessibility to an area due to

the construction of a road, trainees who have found employment); and
— impacts: longer‑term socioeconomic consequences, direct or indirect, positive or negative, that can be

observed a certain period after the completion of an intervention.

## **Executive
 06**

#### **summary**

###### **I**

Within the Commission’s decentralised organisational
framework, the Directorate‑General for Development
and Cooperation — EuropeAid (EuropeAid) has set
up its own results accountability framework which
comprises the monitoring, evaluation and reporting
of its activities. This report examines two of the key
elements of this framework, namely evaluations and
results oriented monitoring (ROM). Evaluation is the
systematic and objective assessment of an ongoing or completed programme or policy, its design,
implementation and results. ROM is a standardised
external review, specific to external aid, designed to
look at programmes’ performance. The main purposes
of these parts of the accountability framework are to
improve the implementation of ongoing programmes
and the design of future programmes and policies
through feedback and lessons learned, and to provide
a basis for accountability.

###### **II**

The Court found that EuropeAid’s evaluation and ROM
systems are not sufficiently reliable.

###### **III**

Overall, EuropeAid’s evaluation and ROM functions are
well organised but there is a lack of overall supervision
of programme evaluation activities. Also, insufficient
attention is paid to the efficient use of evaluation and
ROM resources.

###### **IV**

The evaluation and ROM systems do not sufficiently
ensure that relevant and robust findings are produced.
Programme evaluation plans are based on insuffi­
ciently clear prioritisation criteria. There is no monitor­
ing system to identify and address frequent deviations
from evaluation plans. Quality control procedures are
not implemented consistently for ROM and pro­
gramme evaluations.

###### **V**

The evaluation and ROM systems do not ensure that
maximum use is made of findings because proper
mechanisms are not in place to monitor their fol­
low‑up and dissemination.

###### **VI**

The evaluation and ROM systems do not provide
adequate information on results achieved due to
insufficiently well‑defined objectives and indica­
tors, the limited proportion of _ex post_ evaluations
and ROMs and inherent limitations in the evaluation
methodology for budget support. These factors limit
considerably EuropeAid’s capacity to account for the
actual results achieved.

###### **VII**

The Court provides recommendations on the efficient
use of evaluation and ROM resources, the prioritisation
and monitoring of evaluations, the implementation
of quality control procedures, the demonstration of
results achieved and the follow‑up and dissemination
of evaluation and ROM findings.

## **Introduction 07**

###### **01**

In the current economic and political
context, there is an increased need
for the Commission to ensure and

demonstrate that EU funds are used in
an economic, efficient and effective
manner. The European Parliament and
the Council have indicated a need for

a clearer view of the results achieved [1]
with regard to the EU’s main objec­
tives. Information on results achieved
can only be obtained with a results
accountability framework comprising
three elements: monitoring, evaluation
and reporting.

**The Commission’s**
**evaluation framework**

###### **02**

Evaluation is the systematic and ob­
jective assessment of an ongoing or
completed programme or policy, its
design, implementation and results.
The main purposes are to ascertain
whether the objectives have been
met and to formulate recommenda­
tions with a view to improving future
interventions so as to support deci­
sion‑making by the Commission, the
European Parliament and the Council.

###### **03**

Evaluation is essential for planning,
designing and implementing EU poli­
cies and interventions, and enhancing
transparency and democratic account­
ability. This is emphasised by Article
318 of the Treaty on the Functioning
of the European Union, which requires
the Commission to submit annually
to the European Parliament and the
Council an evaluation report on the
Union’s finances on the basis of the
results achieved (Article 318 evalua­
tion report). As regards the first three
reports produced by the Commission,
in February 2014, the European Parlia­
ment pointed out [2] that they did not
provide the budgetary authority with
a clear view of the real extent to which
the Union’s main objectives have been
achieved.

###### **04**

The Commission’s evaluation frame­
work is set out in a number of policy
and regulatory documents:

(a) the financial regulations applicable

to the EU budget [3] ;

(b) the communication on reinforcing

the use of evaluation [4] ;

(c) the communication on strategic

evaluation [5] ; and

(d) the communication on strengthen­

ing the foundation of Smart Regu­
lation — improving evaluation [6] .

1 For instance, Council
recommendations to the
Commission in relation to the
2011 discharge on ‘getting
results’ (document 5752/13
ADD1, p. 31). European
Parliament’s
recommendations in the
context of the discharge 2011,
see recommendations 248 for
the EEAS and EuropeAid, 308
and 309 for the Commission as
a whole (document
P7_TA‑PROV(2013)0122).

2 Resolution of 26 February 2014
the evaluation of the Union’s
finances based on the results
achieved: a new tool for the
European Commission’s
improved discharge
procedure. Document
P7_TA(2014)0134.

3 Article 30(4) of Regulation (EU,
Euratom) No 966/2012 of the
European Parliament and of
the Council of 25 October 2012
on the financial rules
applicable to the general
budget of the Union (OJ L 298,
26.10.2012, p. 1).

4 SEC(2007) 213 of
21 February 2007:
Communication to the
Commission from
Ms Grybauskaité in agreement
with the President:
‘Responding to Strategic
Needs: Reinforcing the use of
evaluation’.

5 C(2001) 3661 of
12 November 2001:
Communication of the
President ‘Putting evaluation
into practice within the
Commission’.

6 COM(2013) 686 final of
2 October 2013 ‘Strengthening
the foundation of Smart
Regulation — improving
evaluation’.

## **Introduction 08**

###### **05**

Individual directorates-general are
responsible for evaluating their own
activities and can structure their
evaluation systems to suit their needs
and requirements, in compliance with
Commission-defined standards con­
cerning the organisation, the selection,
the design and the use of evaluations
findings.

**EuropeAid’s evaluation**
**and ROM systems**

###### **06**

EuropeAid’s evaluation and results
oriented monitoring (ROM) systems
are part of its results accountability
framework which is based on the pol­
icy and regulatory documents listed
in paragraph 4 as well as in policy and
regulatory documents more specific to
the EU’s development aid:

(a) the financial regulations applicable

to the EDF [7] and the other financing
instruments supporting the EU’s
development policy;

(b) the Paris Declaration, the Accra

Agenda for Action [8] and the Busan
Partnership Agreement [9] ;

(c) the European Consensus on

Development [10] ;

(d) the communication on the ‘Agen­

da for Change’ [11] .

###### **07**

EuropeAid manages two types of
evaluations, which are carried out by
external contractors:

(a) strategic evaluations, which

comprise geographic evaluations
(covering a country or region) and
thematic evaluations, which cover
a sector (e.g. education, food se­
curity) or a method of aid delivery
(e.g. budget support, cooperation
with development banks); and

(b) programme [12] evaluations, that can

be carried out during the imple­
mentation (mid‑term evaluation),
on completion (final evaluation) or
after completion ( _ex post_ evalua­
tion) of an intervention.

###### **08**

These evaluations assess performance
against five criteria developed by the
Development Assistance Committee of
the Organisation for Economic Coop­
eration and Development (OECD‑DAC),
along with the international donor
community [13] : relevance, effectiveness,
efficiency, sustainability and impact.
In addition, evaluations have to assess
the EU’s comparative advantage and
the coherence with other EU policies
and programmes, other donors’ ac­
tions and partner countries’ policies.

7 10th EDF: Articles 12 and 27 of
Council Regulation (EC)
No 215/2008 of
18 February 2008 on the
Financial Regulation
applicable to the 10th
European Development Fund
(OJ L 78, 19.3.2008, p. 1).

11th EDF: Decision No 1/2013
of the ACP‑EU Council of
Ministers of 7 June 2013
adopting a protocol on the
multiannual financial
framework for the period
2014–20 under the Partnership
Agreement between the
members of the African,
Caribbean and Pacific Group
of States of the one part, and
the European Community and
its Member States, of the other
part (OJ L 173, 26.6.2013, p. 67).

8 Paris Declaration on Aid
Effectiveness, 2005 and Accra
Agenda for Action, 2008:
http://www.oecd.org/dac/
effectiveness/parisdeclaratio­
nandaccraagendaforaction.
htm

9 Busan Partnership Agreement
for Effective Development
Cooperation, 2011: http://
www.oecd.org/dac/
effectiveness/fourthhighlevel­
forumonaideffectiveness.htm.

10 Joint statement by the Council

and the representatives of the
governments of the Member
States meeting within the
Council, the European
Parliament and the
Commission on European
Union Development Policy:
‘The European Consensus’
(OJ C 46, 24.2.2006, p. 1).

11 COM(2011) 637 final of

13 October 2011 ‘Increasing
the impact of EU
Development Policy: an
Agenda for Change’.

12 The term _programme_ used in

this report refers to _projects_,
which are small‑scale
interventions, and
_programmes_, which are
large‑scale interventions.

13 http://www.oecd.org/dac/

evaluation/daccriteriaforevalu­
atingdevelopmentassistance.
htm

## **Introduction 09**

###### **09**

The ROM system is used by EuropeAid
to monitor, on the spot, the qual­
ity of its programmes. It is based on
short on‑site visits to programmes by
external experts who rate the pro­
gramme’s performance (see _**Annex I**_ )
in respect of the above mentioned
five OECD‑DAC evaluation criteria
(see paragraph 8). The grading aims
to provide a comparison between
different programmes and for the ag­
gregation of overall portfolio perfor­
mance data. Compared to evaluations,
ROM is a lighter tool which provides
a quick overview of the performance
of programmes. Like programme
evaluations, they can be carried out
during or after the implementation of

programmes.

###### **10**

EuropeAid produces the following key
reports:

(a) the half yearly External Assistance

Management Report (EAMR),
EU Delegations’ main reporting
document [14] ;

(b) the reports of EuropeAid’s direc­

tors about the management of the
funds delegated to them;

(c) the Annual Activity Report (AAR)

submitted by EuropeAid’s Director
General to the Commissioner as
well as to the European Parliament
and the Council;

(d) the Annual Report (AR) on the

European Union’s development
and external assistance policies
submitted by the Commission to
the European Parliament and the
Council and;

(e) the contribution to the Article 318

evaluation report submitted by
the Commission to the European
Parliament and the Council.

14 EAMRs were required annually

from 2014.

## **Audit scope
 10**
#### **and approach**

###### **11**

The Court examined whether Europe­
Aid has reliable evaluation and ROM
systems. The audit focused on three
main questions:

(a) Are EuropeAid’s evaluation and

ROM functions appropriately or­
ganised and resourced?

(b) Do EuropeAid’s evaluation and

ROM systems ensure the provision
of relevant and robust findings?

(c) Do the evaluation and ROM sys­

tems ensure the maximum use of
the findings made?

###### **12**

The audit covered the evaluation and
ROM systems in place in the period
from 2007 to 2013. Where adequate
information was available and could
be checked by the Court, it took ac­
count of changes adopted or planned
by EuropeAid for the period from
2014 to 2020.

###### **13**

The audit was carried out between

November 2013 and March 2014 and

included:

(a) a review of EuropeAid’s evaluation

and ROM guidelines and of the
main documents and processes in
the management of evaluations;

(b) interviews with officials from eight

units at EuropeAid [15] ;

(c) visits to the EU delegations in Be­

nin and India, involving interviews
with the delegations staff and a re­
view of relevant documentation;

(d) a review of programming docu­

ments (see _**Annex II**_ ), programme
financing agreements (see _**An­**_
_**nex III**_ ), strategic and programme
evaluations (see _**Annex IV**_ ), and
ROM reports (see _**Annex V**_ ); and

(e) a survey sent to 50 EU delegations

to which 41 answers were received

(see _**Annex VI**_ ).

15 Evaluation; Quality and

Results; Planning ACP and
Horizontal Coordination;
Geographical Coordination
Asia and Pacific; Education,
Health, Research, Culture;
Rural Development, Food
Security, Nutrition; Africa‑EU
Partnership, Peace Facility;
Regional Programmes
Neighbourhood South.

## **Observations 11**

**Overall, the evaluation**
**and ROM functions are**
**well organised but**
**EuropeAid could pay**
**more attention to the**
**efficient use of resources**

###### **14**

The Court examined whether the
evaluation and ROM systems provide
for a sound distribution of responsi­
bilities and ensure the efficient use of

financial and staff resources.

**There is a sound distribution**
**of responsibilities but**
**insufficient supervision of**
**programme evaluations**

###### **15**

EuropeAid’s organisation successfully
combines centralised and decentral­
ised responsibilities. As regards cen­
trally managed responsibilities:

(a) the Evaluation unit manages stra­

tegic evaluations, which requires
it to have a comprehensive view
of EuropeAid’s activities. This unit
is responsible for developing the
evaluation methodology and for
coordinating and monitoring Euro­
peAid’s evaluation activities. It also
has a central role in the reporting
of strategic evaluations findings in
the AAR and AR and in EuropeAid’s
contribution to the Commission’s
Article 318 evaluation report;

(b) the Quality and Results unit super­

vises the ROM system, including
overall quality assurance com­
mon to the whole organisation,
the coherence between ROM
and programme evaluations, and
contributes to the analysis of ROM
and programme evaluation data.
It provides the ROM aggregated
performance ratings for the AAR
and AR;

(c) ROM coordinators at EuropeAid [16]

supervise the ROM system for their
respective areas of intervention. In
EuropeAid’s operational units and
EU delegations, ROM focal points
provide support to the experts in
organising field visits.

###### **16**

The Evaluation unit and the Qual­
ity and Results unit report directly
to a deputy director‑general. This is
a sufficiently senior position in the or­
ganisation to ensure that they receive
support from senior levels of manage­
ment for their roles.

16 There are five ROM

coordinators in total: one for
geographical programmes in
a region (Africa — Caribbean
— Pacific, Asia, Latin America,
Neighbourhood) and one for
thematic programmes.

## **Observations 12**

###### **17**

The management of programme eval­
uations is decentralised to EuropeAid’s
operational units and EU delegations.
This has potential benefits in terms
of ensuring the take‑up of evaluation
results. However, programme man­
agers have a considerable degree of
autonomy in managing programme
evaluations as they are responsible
at all stages of the process from the
initiation of the evaluations until the
follow‑up of results. In the absence of
overall coordination and monitoring of
this activity EuropeAid has no over­
view and there is little supervision by
senior management.

**EuropeAid does not give**
**enough consideration to the**
**efficient use of resources**

###### **18**

The Commission’s evaluation stand­
ards require that evaluation activities
are appropriately resourced to meet
their purposes. In particular, each
directorate‑general must ensure that
staff and financial resources are clearly
identified and proportionately al­
located for evaluation activities to be

carried out.

###### **19**

EuropeAid has insufficient information
on the resources used by its evalua­
tion and ROM systems. Information is
available as regards the annual cost
of strategic evaluations, which ranges
between 5 and 6 million euro, and
of ROM, which amounted to 17 mil­
lion euro over the 2007–13 period [17] .
According to estimates made in
2010 and 2013 by EuropeAid, the
annual cost of programme evalua­
tions ranges between 12 and 15 mil­
lion euro. The total annual cost of

evaluations and ROM would thus

amount to 34–38 million euro. Howev­
er, as EuropeAid has no reliable infor­
mation on the number of programme
evaluations and since their cost cannot
be easily identified in the IT manage­
ment information system [18], the Court
cannot confirm the accuracy of these
estimates.

###### **20**

There are 16 staff in the Evaluation unit
and about one full‑time equivalent
staff member involved in ROM at the
Quality and Results unit. But there is
no management information on staff
resources involved in programme
evaluations and ROM reviews at the
level of programme managers, ROM
focal points and ROM coordinators. On
the basis of its survey, the Court esti­
mates that there are approximately 60
full time equivalent staff dealing with
ROM and programme evaluations in
the EU delegations. The total number
of full-time equivalent staff dealing
with evaluations and ROM is thus likely
to be well above 80.

17 EuropeAid estimates that this

annual cost will increase to
about 25,5 million euro, due in
particular to longer on‑site
visits (from an average of
2 days to 8 days) and the use
of more senior experts.

18 Common Relex Information

System.

## **Observations 13**

###### **21**

Due to the lack of reliable information
EuropeAid cannot ensure that staff
and financial resources are appropriate
and proportionately allocated to the
various evaluation and ROM activities.

###### **22**

The Court found inadequate coordina­
tion between programme evaluations
and ROMs, which is likely to lead to
inefficiencies. Programme evaluations
and ROMs follow completely different
planning approaches and criteria [19] (see
paragraphs 26 and 30). Programmes
can be over- or under-assessed due
to an inappropriate combination of
ROM and evaluations, or sometimes
not assessed at all, as is frequently the
case at the post completion stage (see
paragraphs 43 to 46). This is confirmed
by the Court’s survey: only half of
the replies consider that programme
evaluations and ROMs complement
each other well.

**EuropeAid’s evaluation**
**and ROM systems do not**
**sufficiently ensure**
**relevant and robust**
**findings**

###### **23**

The Court examined whether evalu
ations and ROMs were selected on the
basis of appropriate criteria and are
carried out on time. Furthermore, it
examined whether they were carried
out by sufficiently qualified experts,
whether quality control procedures
were applied effectively and whether
they provided sufficient evidence on
the results achieved by EuropeAid’s
interventions.

**There are some weaknesses**
**in the selection of**
**programme evaluations**
**and timely delivery of**
**evaluations**

**Multiannual plans for strategic**
**evaluations are based on**
**appropriate criteria**

###### **24**

The indicative multiannual evaluation
plan for the 2007–13 period aimed
to achieve full geographic coverage
with a view to drawing lessons for the
subsequent programming process.
Thematic evaluations were selected
on the basis of the materiality of funds
allocated and the importance of sec­
tors in country‑level and regional EU
cooperation strategies. The objective
was also to increase the number of
evaluations of aid delivery methods
(e.g. budget support, cooperation with
the EIB, development banks, coopera­
tion with UN organisations).

19 The only rule is that there

should not be a ROM the same
year a programme evaluation
is carried out.

## **Observations 14**

###### **25**

For the rolling 2014–18 work pro­
gramme, the selection criteria were
further refined. The objective was to
provide sufficient material to report,
in 2017, to the European Parliament
and the Council on the results and
impacts of the financial instruments,
as required by the common rules and
procedures for the implementation of
the Union’s instruments for financing
external action [20] . Thematic evaluations
were selected on the basis of priority
themes of the Agenda for Change [21],
and the aim was to ensure a propor­
tional balance between regions (Africa,
Latin America and Asia) and types of
countries (middle‑income countries,
fragile states and countries with sig­
nificant budget support programmes).

**The criteria for programme**
**evaluation selection are unclear**

###### **26**

The main reason for carrying out
a programme evaluation is that it is
an obligation or a possibility set out
in the financing agreement signed
with the beneficiary or implement­
ing partner. The provision for evalu-​
ations in these agreements is decided
on a case‑by‑case basis at the design
phase of the programme but no guid­
ance is given on how to make this deci­
sion. The Court’s review of 20 recent
programmes (see _**Annex III**_ ) identified
significant and unexplained differ­
ences in approach:

(a) mid‑term evaluations were provid­

ed for in all programmes but were
compulsory in only half of them;

(b) _ex post_ evaluations were provided

for in only two thirds of the pro­
grammes and were compulsory in
only one of them.

**Evaluations are often not**
**delivered as planned**

###### **27**

EuropeAid has not set up a proper
system to monitor the implementation
of strategic and programme evaluation
plans, analyse the reasons for devia­
tions, and address the systemic causes
of delay.

###### **28**

Strategic evaluations often took much
more time than planned [22] . This con­
tributed to the incomplete execution
of the indicative multiannual evalua­
tion programme for the 2007–13 per-​
iod; only 70 % of the selected evalu­
ations were completed. According to
EuropeAid, this was particularly due to
staff resource constraints in the Evalu­

ation unit.

20 Article 17 of Regulation (EU)

No 236/2014 of the European
Parliament and of the Council
of 11 March 2014 laying down
common rules and procedures
for the implementation of the
Union’s instruments for
financing external action
(OJ L 77, 15.3.2014, p. 95).

21 Key policy priorities are:

  - Human rights, democracy,

governance (including rule
of law, gender equality and
the empowerment of
women, public sector
management, tax policy
and administration,
corruption, civil society and
local authorities, natural

resources,
development‑security);

   - Inclusive and sustainable

growth (social protection,
health, education and jobs,
business environment,
regional integration and
world markets, sustainable
agriculture and energy).

22 Country evaluations took on

average 18 months instead of
the planned 12 months.

## **Observations 15**

###### **29**

Even though programme evaluations
are usually carried out according to
their timetable, the Court’s survey
found that, in the period 2010–13,
there was a significant difference
between the evaluations planned
and carried out [23] . The provisions set
out in the financing agreements were
not systematically applied during
programme implementation as evalu-​
ations were sometimes postponed or
cancelled. The Court’s survey indi­
cates that EU delegations take such
decisions on the basis of factors like
limited staff resources [24], the needs of
evaluation users and problematic pro­
gramme implementation. The reasons
for not carrying out planned evalu­
ations were not documented. This

increases the risk that decisions not to
carry out an evaluation are not taken
for sound reasons, or at the appropri­
ate level.

**The ROM selection**
**methodology reduces the**
**usefulness of ROM**

###### **30**

The selection of ROM reviews is based
on clearly defined standard criteria [25]
and is made in consultation between
the ROM contractors and programme
managers. ROM reviews are usually
carried out according to plan and
reports are usually delivered without
significant delays.

###### **31**

However, in its Special Report
No 1/2011 [26] the Court pointed to key
limitations which reduce the useful­

ness of ROM as an indicator for the
overall aid portfolio, notably that
the sample of projects selected is
not established on a statistical but
rather on a judgemental basis. In 2011,
EuropeAid’s Internal Audit Capabil­
ity also found that, as the selection is
not risk‑based, ROM reviews do not
provide the feedback to programme
managers where it is most needed to
improve programme implementation.

**The procedures to ensure**
**the quality of programme**
**evaluations and ROMs are**
**implemented inconsistently**

**Contracting procedures ensure**
**the recruitment of sufficiently**
**qualified experts**

###### **32**

Evaluations are carried out by external
consultants contracted by EuropeAid
through public procurement proce­
dures. Proposals submitted by consul­
tancy firms are scored on the basis of
specific requirements [27], in particular
those concerning the experts’ profes­
sional qualification. Team leaders and
senior experts usually have 10 to 15
years of professional experience in
the field of intervention, plus 5 years
of experience in evaluation for team
leaders.

23 On average, 3,5 annual

evaluations were planned by
EU delegations, and
2,9 evaluations were
performed.

24 Mentioned as the main reason

for not carrying out a planned
evaluation.

25 The main reasons for

departing from the standard
criteria are: (i) to avoid
unnecessary monitoring of
well‑managed programmes or
programmes evaluated
recently; (ii) to cover
problematic programmes
which fall below the financial
threshold; and (iii) to
strengthen monitoring efforts
on programmes with
insufficient internal
monitoring.

26 Special Report No 1/2011 ‘Has

the devolution of the
Commission’s management of
external assistance from its
headquarters to its
delegations led to improved
aid delivery’, paragraph 25
(http://eca.europa.eu).

27 Terms of reference or technical

specifications.

## **Observations 16**

###### **33**

ROM reviews are also carried out by
external consultancy firms (ROM con­
tractors) selected through public pro­
curement procedures [28] . The required
experience of development coopera­
tion in the given region is 10 years
for senior experts, 5 to 10 years for
mid‑level experts and up to 5 years
for junior experts. For contracts to be
signed in 2014, EuropeAid required
ROM contractors to hire specialised
experts on long-term assignments in
order to further enhance the level of

expertise.

**Quality control procedures for**
**programme evaluations and**
**ROMs are not systematically**
**applied**

###### **34**

The Commission’s evaluations stand­
ards stipulate that, save in duly justi­
fied cases, a reference group [29] must
be set up for each evaluation to advise
on the terms of reference, support
the evaluation work and take part in
assessing the quality of the evaluation
with appropriate regularity. Europe­
Aid’s evaluation guidelines stress that
quality assurance at each step of the
process is a key task of the evaluation
manager, together with the reference
group, which should include repre­
sentatives of the main stakeholders

concerned [30] .

###### **35**

Reference groups are systematically
set up for strategic evaluations. They
meet on a regular basis and their work,
including quality review at all stages, is
documented in meeting minutes.

###### **36**

This was not the case for programme
evaluations. According to the Court’s
survey, only 39 % of EU delegations
set up a reference group always or
most of the time [31], and a reference
group was rarely formally set up by
EuropeAid’s operational units and EU
delegations visited by the Court. Some
consultations may take place between
the programme manager and other
stakeholders at various stages of the
evaluation process (e.g. for drafting
the terms of reference or reviewing the
draft evaluation report). However, in
the absence of appropriate documen­
tation, it was not possible for the Court
to assess if this was frequent practice.

###### **37**

EuropeAid’s evaluation guidelines
provide that the quality of the draft
and final evaluation reports must be
assessed on the basis of nine well‑de­
fined criteria [32] . For this purpose,
a quality grid must be filled by two
persons who discuss the nine criteria.
For strategic evaluations, the qual­
ity grid was filled systematically and
published together with the evalu-​
ation report on EuropeAid’s website [33] .
For programme evaluations, according
to the Court’s survey, only 71 % of EU
delegations assess the quality of the
evaluation report and only 37 % fill the
assessment grid [34] .

28 There are five ROM

contractors in total: one per
region for geographical
programmes (Africa —
Caribbean — Pacific; Asia;
Latin America;
Neighbourhood) and one for
the thematic programmes.

29 Also referred to as ‘steering

group’ in the Commission.

30 Depending on the type and

place of the evaluation, the
reference groups may include
representatives from
EuropeAid, the EEAS, other
Commission’s departments,
partner country’s authorities,
the civil society organisations,
other donor agencies, or other
experts.

31 No formal reference groups

were created in the
delegations and operational
units visited during the audit
except in the case of
programme evaluations
similar to strategic evaluations.

32 The nine criteria are: (1)

Meeting needs; (2) Relevant
scope; (3) Defensible design;
(4) Reliable data; (5) Sound
analysis; (6) Credible findings;
(7) Valid conclusions; (8) Useful
recommendations; (9) Clear
report. For programme
evaluations, criteria 2 and 3 are
combined into one:
Appropriate design. Each
criteria is rated by five levels:
excellent, very good, good,
poor and unacceptable.
(http://ec.europa.eu/
europeaid/evaluation/
methodology/guidelines/
gui_qal_flr_en.htm#03).

33 http://ec.europa.eu/

europeaid/evaluation‑
reports_en

34 This grid was filled in for only

one out of the ten programme
evaluations reviewed by the
Court, the African Peace
Facility evaluation.

## **Observations 17**

###### **38**

Programme managers are required to
assess the quality of ROM reports by
filling a compulsory response sheet.
This was done for less than 60 % of
ROM reports delivered over the period
from 2011 to 2013.

**The evaluation and ROM**
**systems do not provide**
**adequate information on**
**results achieved**

**Programme objectives and**
**indicators are not well defined**

###### **39**

To enable an evidence‑based assess­

ment of achievements to be carried
out, there must be specific, measurable and time‑bound objectives for
programmes with objectively verifi­
able indicators. The objectives men­
tioned in high level policy documents
and legal texts establishing the financ­
ing instruments are prepared in a po­
litical context and, while they should
provide a basis, are often not focused
enough to be useful at implementing
level. They need to be translated into
more specific operational objectives
when programming aid for a country,
region or sector, and designing indi­
vidual interventions.

###### **40**

Out of the 15 thematic, regional and
country‑level programming docu­
ments reviewed by the Court (see
_**Annex II**_ ), five [35] contained clearly
described objectives and indicators,
but set no baselines or targets against
which to assess performance. For the
other 10, objectives were unclear and
indicators often lacking (see _**Box 1**_ ).

###### **41**

Concerning individual programmes,
several Court’s reports [36] have high­
lighted that objectives are insuffi­
ciently clear and that no measurable
indicators are defined. Out of the 10
programme evaluation reports re­
viewed by the Court, four mentioned
unclear objectives and indicators, as
well as the absence of baselines and
unavailability of monitoring data, as
major weaknesses [37] (see _**Box 2**_ ). Only
two evaluations made positive com­
ments in this respect [38] .

###### **42**

In the absence of clearly defined
objectives and indicators, it is not
possible for an evaluation to establish
evidence-based performance assess­
ments. This issue was also mentioned
in a 2011 study commissioned by
EuropeAid [39] .

35 Food Security, Benin, Chad,

Uruguay and Zambia.

36 E.g. Special Report No 1/2012

‘Effectiveness of European
Union development aid for
food security in sub‑Saharan
Africa’, paragraph 51 and
Special Report No 13/2013 ‘EU
Development Assistance to
Central Asia’, paragraph 52.
(http://eca.europa.eu).

37 Similar comments were made

by two ROM contractors, one
responsible for the ROM, for
the European Neighbourhood
Partnership Instrument and
the other responsible for the
ROM for the Centrally
Managed Thematic
Programmes, in their annual
reports 2011, 2012 and 2013.

38 Benin: PACTE — programme

d’appui aux collectivités
territoriales ; Food Facility
programmes implemented
with the African Union
— Interafrican Bureau for
Animal Resources (AU‑IBAR):
Livestock Emergency
Intervention to mitigate Food
Crisis in Somalia (LEISOM) and
Vaccines for the Control of
Neglected Animal Diseases in
Africa (VACNADA).

39 Study on Legal Instruments

and Lessons Learned from the
Evaluations Managed by the
Joint Evaluation Unit, July
2011, paragraph 26, p. ix (it
does not cover the EDF).
http://ec.europa.eu/
europeaid/
evaluation‑reports_en, under
the section ‘per instrument /
channel’.

## **Observations 18**

**Examples of vague objectives in multiannual programming documents**

**Investing in People: Mid‑term Review of Strategy Paper for Thematic Programme (2007–13) and**
**Multiannual Indicative Programming 2011–13**

As regards the strengthening of health systems and universal access to basic health care:

‘In terms of delivery, the thematic action should focus on supporting activities that:

οο stimulate effective leverage of resources for inclusive health system strengthening and universal coverage
of health service delivery;

οο promote and support the application of innovative ways and good practices to address weaknesses in
health service delivery and to improve quality of care;

οο support the role of civil society, professional associations and other stakeholders in their advocacy, policy
monitoring and advisory role in the area of health;

οο support platforms for policy dialogue and exchange of good practices on different aspects of health at
regional and global levels;

οο support partner countries in the process of implementing international legal commitments in the area of
health.’

**CSP/NIP Bolivia 2007–13**

_**‘Strengthening the productive sector comprising micro‑enterprises and SMEs, and improving the institutions**_
_**framework in order to increase their productivity and competitiveness, which will lead to better market access**_
_**and integration with the purpose of creating decent and sustainable work.’**_

**Examples of vague objectives in programmes**

**EuroMed Migration II 2008–11**

_**‘Strengthen cooperation in the management or migration so as to build the MEDA partners’ capacities to pro­**_
_**vide an effective, targeted and comprehensive solution for the various forms of migration.’**_

**African Peace Facility 2003–13**

_**‘Support the operationalisation of the African Peace and Security Architecture (APSA) and to increase the**_
_**effectiveness of the African Union and the African regional organisations on the conduct of peace support**_
_**operations in Africa.’**_

## **Observations 19**

**Evaluations and ROMs focus**
**more on the implementation**
**of programmes than on results**
**achieved**

###### **43**

EuropeAid’s evaluation guidelines
indicate that:

(a) mid‑term and final evaluations

are intended to draw lessons from
the first years of the intervention
implementation, adjust the ongo­
ing intervention where necessary
and provide an analysis of the first
results and impacts achieved;

(b) _ex post_ evaluations are mainly

concerned with checking achieved
impacts and assessing the sustain­
ability of the intervention’s ben­
efit, report to the institutions that
have allocated the resources and
help to transfer acquired experi­
ence to other countries or sectors.

###### **44**

_Ex post_ programme evaluations are
seldom carried out. For the two EU
delegations visited, only three out of
the 17 evaluations carried out in the
2010–13 period were _ex post_ evalu-​
ations. Financing agreements rarely set
an obligation to perform such evalu­
ations (see paragraph 26(b)). Another
reason for the lack of _ex post_ evalu-​
ations is that programme evaluations
are funded from the programme’s own
budget and are therefore constrained
by its completion date.

###### **45**

According to EuropeAid, additional
contributing factors are the una­
vailability of monitoring data after
programme completion and the
possible evolutions in the partner
country’s context that may affect the
programme’s effectiveness. The Court
does not consider these to be valid
arguments: the monitoring system
can be designed to continue collect­
ing the data deemed necessary during
a certain period after programme
completion (see paragraph 47) and
the purpose of evaluation is also to
identify and assess the impact of unex­
pected factors.

###### **46**

The ROM guidelines [40] indicate that _ex_
_post_ ROMs allow solid information to
be gathered about a project’s design,
real impact and sustainability which
can only be fully appreciated after
a project has ended. But _ex post_ ROMs
represented on average only 11 % of
the ROMs carried out in the 2008–12
period. EuropeAid decided to discon­
tinue with them from 2014. It consid­
ered that the methodology for _ex post_
ROM was inappropriate for assessing
impact and that the revised ROM sys­
tem aims only to support programme
management and implementation.

40 See ROM Handbook 2012,

p. 57.

## **Observations 20**

###### **47**

In contrast with the limited propor­
tion of _ex post_ programme evaluations
and ROMs, the Results Measurement
framework (ReM) implemented at the
European Investment Bank (EIB) since
2012 includes _ex post_ monitoring until
3 years after project completion and
the EIB intends to use the data thus
collected for _ex post_ evaluations.

**The methodology for**
**evaluating budget support has**
**inherent limitations**

###### **48**

Under the lead of EuropeAid’s Evalu­
ation unit, the OECD DAC Network on
Development Evaluation produced
a document called ‘Evaluating Budget
Support — Methodological Approach’
which was published in 2012 [41] . It is
a common evaluation framework to
evaluate budget support operations
and provides guidelines on how to or­
ganise and carry out these evaluations.
The document recognises that in most
cases, it will be difficult to quantify the
budget support contribution, which
will need to be assessed in a qualita­
tive manner.

###### **49**

While qualitative assessments are, of
course, considered in all evaluations,
the difficulty in establishing unam­
biguous logical cause‑effect links
between results observed and the
budget support programmes weakens
the reliability of evidence provided by
the evaluation. This inherent limitation
is not mentioned either in budget sup­
port evaluation reports [42] or in Europe­
Aid’s reports [43] .

**The reporting does not provide**
**much evidence on results**
**achieved**

###### **50**

EuropeAid’s capacity to assess and re­
port on results achieved is significantly
reduced by insufficiently clear objec­
tives and indicators set for operational
programmes and by the focus put on
evaluations and ROM during the im­
plementation of programmes.

###### **51**

As the Court mentioned in several
reports [44], there is little information on
the actual results of the aid in the EU
delegations’ EAMRs and EuropeAid’s
two key annual reports which tend to
focus more on activities implemented,
rather than on policy achievements.

###### **52**

This is also the case with EuropeAid’s
contribution to the Commission’s
Article 318 evaluation reports. In its
2012 annual report [45], the Court found
that, despite some positive develop­
ments, these reports did not provide
sufficient, relevant and reliable evi­
dence on what the EU’s policies have
achieved and that they were therefore
not suitable for their intended use in
the discharge procedure.

41 http://www.oecd.org/dac/

evaluation/dcdndep/
Methodological%20
approach%20BS%20
evaluations%20Sept%20
2012%20_with%20cover%20
Thi.pdf

42 EuropeAid has managed four

budget support evaluations:
Mali and Tunisia (2011),
Tanzania and South Africa
(2013).

43 AAR 2011, AAR 2013 and AR

2012.

44 Annual Report of the Court of

Auditors on the
implementation of the budget
concerning the financial year
2011, paragraph 10.12 (OJ
C 344, 12.11.2012); Special
Report No 1/2011,
paragraph 59; Special Report
No 13/2013, paragraphs 78 to
80 (http://eca.europa.eu).

45 Annual Report of the Court of

Auditors on the
implementation of the budget
concerning the financial year
2012, paragraphs 10.21 and
10.22 (OJ C 331, 14.11.2013).

## **Observations 21**

**EuropeAid’s evaluation**
**and ROM systems do not**
**ensure the maximum use**
**of the findings made**

###### **53**

The Court examined whether Euro­
peAid’s evaluation and ROM systems
ensured that the findings made were
properly followed up and disseminat­
ed to all intended users.

**Proper follow‑up**
**mechanisms are not in place**

**There are weaknesses in**
**the follow‑up of strategic**
**evaluations findings**

###### **54**

Actions to be taken following a stra­
tegic evaluation are described in
a document [46] published together with
the evaluation report on EuropeAid’s
website. While this is sound practice,
most of such documents reviewed by
the Court did not provide sufficiently
detailed information on the planned
actions, their timetable and the alloca­
tion of responsibilities.

###### **55**

The follow‑up of the action plan by
the Evaluation unit is limited to 1 year
after the publication of the evalua­
tion report. This is often too short to
implement actions plans which involve
amendments to sectorial policies and
country or regional EU cooperation
strategies [47] .

###### **56**

In 2012, an OECD DAC Peer Review [48]
also found that EuropeAid’s Evaluation
unit did not have a clear overview of

the extent to which recommendations
made on the strength of strategic
evaluations were accepted and used to
inform new policies and programmes.
In 2014, a study commissioned by the
Evaluation unit on the uptake of strate­
gic evaluation results found that they
were not used effectively.

46 Called ‘fiche contradictoire’.

47 For example, the thematic

evaluation of the European
Commission support to
respect of Human Rights and
Fundamental Freedoms was
issued in 2011. Many actions
were to be completed after
2012. This was the case for
most evaluations completed
in 2012 and 2013, for which
recommendations were to be
implemented mostly during
the programming period
2014–20.

48 European Union (2012), DAC

Peer Review: Main Findings
and Recommendations
http://www.oecd.org/dac/
peer_reviews/europeanunion­
2012dacpeerreviewmainfind­
ingsandrecommendations.
htm

## **Observations 22**

**There is a lack of documented**
**follow‑up for programme**
**evaluations and ROM findings**

###### **57**

Programme evaluation and ROM find­
ings are usually discussed within op­
erational units or EU delegations and
programme managers are responsible
for ensuring that appropriate actions
are taken. Unlike for strategic evalu­
ations, and as the Court found in the
two EU delegations visited, documen-​
ted action plans are not systematically
prepared and monitored. According
to the Court’s survey, only 44 % of EU
delegations usually draw up action
plans. Concerning ROM reviews, the
response sheets reviewed by the Court
described actions to be taken. But
response sheets were completed for
only 60 % of ROM reviews carried out
in the period from 2011 to 2013.

###### **58**

In more than half of the cases re­
viewed [49], the Court found evidence of
use being made of some findings of
the programme evaluations and ROMs.
In the absence of an appropriate
follow‑up system, EuropeAid cannot
ensure that findings and recommenda­
tions from programme evaluations and
ROM reviews are used effectively.

**Evaluation and ROM findings**
**are well disseminated except**
**for programme evaluations**

**Dissemination is carried**
**out properly for strategic**
**evaluations and ROM findings**

###### **59**

Strategic evaluation and ROM reports
are easily accessible to all EuropeAid’s
units and EU delegations. Strategic
evaluation reports are also available
to external stakeholders. They are
published on the websites of Europe­
Aid and the OECD, and are sent to the
European Parliament, the Council and
EU Member States. The reports and
lessons learned are also disseminated
in a variety of ways (e.g. meetings,
workshops and seminars) adapted to
the target audience (e.g. Commission
departments, EEAS, EU delegations,
European institutions, external part­
ners, and networks of experts).

###### **60**

EuropeAid presents a synthesis of
strategic evaluation results and the
key lessons learned, as well as the ag­
gregated ROM performance data in its
AAR [50] and AR [51] . It also contributes to
the preparation of the Commission’s
Article 318 evaluation report which
provides summaries of strategic evalu­
ation findings [52] .

49 For six of the ten programme

evaluations and six out of the
eight ROM reviews examined.

50 AAR 2013, pp. 117 to 119 and

150 to 162, and Annex 9.

51 Annual report on the

European Union’s
development and external
assistance policies and their
implementation in 2012,
pp. 166 to 173.

52 COM(2013) 461 final of

26 June 2013, Report from the
Commission to the European
Parliament and the Council on
the evaluation of the Union’s
finances based on the results
achieved, pp. 22 to 23.

## **Observations 23**

**Dissemination of programme**
**evaluations is inadequate**

###### **61**

Programme evaluation reports are
generally shared and discussed by
programme managers with directly
interested stakeholders, i.e. mainly Eu­
ropeAid’s geographical and thematic
units and the implementing partner.
They are rarely shared with beneficiar­
ies and other donors.

###### **62**

Delegations are required to report
on lessons learned from programme
evaluations in their half‑yearly EAMRs
to EuropeAid. EAMRs reviewed by the
Court contained no or little informa­
tion in this respect. For programme
evaluations managed by EuropeAid’s
operational units, there was no similar
reporting requirement until 2012. In
EuropeAid’s 2013 AAR [53] a summary of
15 of these programme evaluations is
provided; however, what this repre­
sents in terms of total evaluations
completed in 2013 is not indicated.

###### **63**

As the Court has pointed out [54], pro­
gramme evaluation reports cannot be
systematically used because there is
no central database for holding such
evaluations. This reduces significantly
EuropeAid’s capacity for analysing and
disseminating this information inter­
nally and to external stakeholders and
for providing a basis for accountability.

###### **64**

EuropeAid has stated its intention to
develop a new IT tool to manage, store
and analyse ROM and programme
evaluations, which would serve as
a central storage point for all reports
and allow users to perform advanced
searches and analysis. At the time
of the Court’s audit, the decision to
develop the part of the application
for programme evaluations had not
yet been made, even though report ac­
cessibility was a more crucial issue for
programme evaluations.

53 AAR 2013, Annex 9:

Performance information
included in evaluations.

54 Special Report No 1/2011,

paragraph 60.

## **Conclusions and 24**

#### **recommendations**

###### **65**

The Court concludes that EuropeAid
evaluation and ROM systems are not
sufficiently reliable.

###### **66**

Overall, the evaluation and ROM
functions are well organised, with an
appropriate combination of central­
ised and decentralised responsibilities
which aim to promote a culture of
evaluation and give evaluation a prom­
inent role in the aid management
process. However, there is inadequate
overall supervision of programme
evaluations by senior management. In
the absence of proper management
information on staff and financial
resources used for programme evalu­
ations, EuropeAid cannot ensure that
they are appropriate and are efficiently
allocated to the various evaluation

activities.

###### **67**

The evaluation and ROM systems do
not sufficiently ensure the delivery of
relevant and robust evaluations and
ROM findings. Evaluation plans, drawn
up on the basis of insufficiently clear
criteria, do not guarantee that prior­
ity is given to those evaluations which
are most useful for decision‑making.
In addition, the absence of monitoring
prevents EuropeAid from identifying
and addressing the causes of frequent
deviations from these plans. Procure­
ment procedures enable external
experts with the required skills and
knowledge to be selected. Quality
control procedures for ROM and pro­
gramme evaluations are not applied
consistently within EuropeAid.

###### **68**

Maximum use is not made of evalu­
ations and ROM findings. Proper
follow‑up mechanisms are not in place
to ensure that evaluation and ROM
findings are appropriately used. Strate­
gic evaluations and ROM findings are
widely disseminated through a variety
of communication tools to a wide
range of internal and external stake­
holders. However, the lack of access
to programme evaluations and the
absence of search tools are obstacles
to the proper sharing of knowledge.

###### **69**

The evaluation and ROM systems do
not provide adequate information on
results achieved due to insufficiently
well‑defined objectives and indica­
tors, the limited proportion of _ex post_
evaluations and ROMs and inherent

limitations in the evaluation method­
ology for budget support. As a result,
EuropeAid’s capacity to account for
its activities is limited and its report­
ing gives few indications of the actual
results achieved.

###### **70**

Recently, the Commission and Euro­
peAid launched initiatives to improve
their capacity to assess and report on
policy achievements. These initiatives
were at too early a stage for the Court
to assess whether they were effective­
ly addressing the issues raised in the
present report.

## **Conclusions and recommendations 25**

**Recommendation 1**
**Efficient use of resources**

**for evaluation and ROM**

**systems**

EuropeAid should maintain adequate
management information and perform
needs assessments on a regular basis
in order to ensure an informed alloca­

tion of financial and staff resources
between programme evaluations and
ROM.

**Recommendation 2**
**Appropriate prioritisation of**
**programme evaluations and**

**monitoring of evaluation**

**activities**

In order to ensure that evaluations
carried out reflect the priorities of the
organisation, EuropeAid should:

— define clear selection criteria to

prioritise programme evaluations
and document how they were
applied in establishing the evalua­
tion plans, taking into account the
complementarity with ROM;

— significantly strengthen its system

to monitor and report on the im­
plementation of evaluation plans,
including an analysis of the rea­
sons for delays and the description
of measures adopted to address
them;

— strengthen the overall supervision

of programme evaluation activities
by EuropeAid.

**Recommendation 3**

**Quality control**

In order to ensure the quality of
programme evaluations and ROMs,
EuropeAid should:

— insist that operational units and

delegations apply the quality
control requirements, including for
programme evaluations the use of
a reference group and documenta­
tion of quality controls performed;

— check, on a regular basis, the ap­

plication of these controls.

**Recommendation 4**
**Demonstrating results**

**achieved**

In order to enhance the evaluation
system’s capability to provide ade­
quate information on results achieved,
EuropeAid should:

— apply more rigorously the regula­

tory provisions requiring the use
of SMART objectives and verifiable
indicators;

— modify the monitoring system

so that it continues to provide
data on programmes until at least
3 years after their completion; and

— increase significantly the pro­

portion of _ex post_ programme
evaluations.

## **Conclusions and recommendations 26**

**Recommendation 5**

**Follow‑up and**
**dissemination of evaluation**

**and ROM findings**

In order to ensure the maximum use of
evaluation and ROM findings, Europe­
Aid should:

— extend the follow‑up period for

strategic evaluations;

— develop a central database for

programme evaluations to store
evaluations reports, action plans
and their follow‑up, with search
functionalities which permit
analysis.

This Report was adopted by Chamber III, headed by Mr Karel PINXTEN, Member
of the Court of Auditors, in Luxembourg at its meeting of 21 October 2014.

_For the Court of Auditors_

Vítor Manuel da SILVA CALDEIRA

_President_

## **Annexes 27**

**ROM grading system**

ROM assesses a programme performance in respect of the five evaluation criteria (relevance, efficiency, effec­
tiveness, impact prospects and potential sustainability) according to the following grading system:

|Grading|Numerical|Qualitative|Explanation for the grade|
|---|---|---|---|
|A|4|Very good|The situation is considered highly satisfactory, largely above average and potentially<br>a reference for good practice. Recommendations focus on the need to adopt these<br>good practices in other operations.|
|B|3|Good|The situation is considered satisfactory, but there is room for improvements. Recom­<br>mendations are useful, but not vital for the operation.|
|C|2|Problems|There are issues which need to be addressed; otherwise the global performance of<br>the operation may be negatively affected. Necessary improvements however do not<br>require major revisions of the operations’ strategy.|
|D|1|Serious deficiencies|There are deficiencies which are so serious that, if not addressed, they can lead to<br>failure of the operation. Major adjustments and revision of the strategy are necessary.|

Each programme is then categorised, according to the number of A, B, C and Ds it has received. There are four
categories in which a programme can be classified. ‘I’ indicates very good performance; ‘II’ indicates good per­
formance; ‘III’ indicates that a programme is performing with problems; and ‘IV’ indicates that a programme is
not performing, or that it is experiencing major difficulties.

EuropeAid then determines the percentage of programmes in the ROM portfolio falling under each category.
This is done for the whole portfolio, per region and per sector within a region. This is the basis for its reporting
on the portfolio performance.

## **Annexes 28**

**List of programming documents reviewed**

**Thematic**

**ο** ο Environment and Natural Resources Thematic Programme (ENRTP): strategy paper and multiannual indicative
programme (MIP) 2011–13

**ο** ο Food security thematic programme, strategy paper and MIP 2011–13

**ο** ο Investing in people: Mid‑term Review of strategy paper for Thematic Programme 2007–13 and MIP 2011–13

**Regional**

**ο** ο Central Asia Regional Strategy Paper 2007–13

**ο** ο Latin America Regional Indicative Programme 2007–13

**ο** ο Asia MIP 2007–10

**ο** ο Intra ACP cooperation 10th EDF, MIP 2008–13

**Country**

**ο** ο Country Strategic Paper (CSP) Laos 2007–13

**ο** ο CSP Bangladesh 2007–13

**ο** ο CSP India 2007–13

**ο** ο CSP/National Indicative Programme (NIP) Benin 2008–13

**ο** ο CSP/NIP Chad 2008–13

**ο** ο CSP/NIP Zambia 2008–13

**ο** ο CSP/NIP Uruguay 2007–13

**ο** ο CSP/NIP Bolivia 2007–13

## **Annexes 29**

**List of financing agreements reviewed**

(euro)

|Country|Decision number|Title|EU<br>contribution|
|---|---|---|---|
|**Africa**|**Africa**|**Africa**|**Africa**|
|Madagascar|FED/2012/023-503 (EC)|Amélioration de la Sécurité Alimentaire et Augmentation des Revenus<br>Agricoles (ASARA)|40 000 000|
|Uganda|FED/2013/023-513 (EG)|Small and Medium Agribusiness Development Fund — A Public–Private<br>Partnership.|15 000 000|
|Ethiopia|FED/2011/023-531 (CA)|Capacity development and governance support for scaling up renewable en­<br>ergy services in rural and peri-urban areas of Ethiopia — CfP 129364 Prop 353|1 176 000|
|Somalia|FED/2012/023-536 (EC)|Support to Democratisation|14 000 000|
|Papua New Guinea|FED/2012/023-538 (EG)|Non State Actors (NSAs) Support Programme in Papua New Guinea|4 538 000|
|Nigeria|FED/2013/023-551 (EG)|Energising Access to Sustainable Energy in Nigeria (EASE)|27 000 000|
|Kenya|FED/2012/023-566 (EC)|Standards for Market Access Programme (SMAP)|12 100 000|
|Liberia|FED/2012/023-568 (EC)|Support to trade facilitation in the customs sector|3 000 000|
|||**8 Financing Agreements** <br>**TOTAL**|**116 814 000**|
|**Latin America**|**Latin America**|**Latin America**|**Latin America**|
|Honduras|DCI-ALA/2013/023-720 (EC)|Promoting fair and accessible justice in Honduras (EuroJusticia)|27 500 000|
|Costa Rica|DCI-ALA/2010/021-501 (EC)|Comprehensive strategy to reduce school drop-out rates in public secondary<br>education|8 500 000|
|Dominican Republic|BAN/2012/024-100 (EC)|Banana accompanying measure (BAM) Dominican Republic|16 340 000|
|Uruguay|DCI-ALA/2011/022-498 (EC)|Support for the reform of the Uruguayan penal justice and penitentiary<br>systems for the improvement of the quality of life and social and labour<br>reintegration of prison inmates|5 000 000|
|||**4 Financing Agreements** <br>**TOTAL**|**57 340 000**|
|**Neighbourhood**|**Neighbourhood**|**Neighbourhood**|**Neighbourhood**|
|Egypt|ENPI/2012/024-080 (EC)|Emergency Employment Investment Project|70 000 000|
|Morocco|ENPI/2012/024-196 (EC)|Programme d’appui à l’Initiative Nationale de Développement Humain (INDH)<br>— phase II|25 000 000|
|Tunisia|ENPI/2011/022-772 (EC)|Programme de Modernisation des services|20 000 000|
|West Bank and Gaza<br>Strip|ENPI/2010/022-476 (EC)|PEGASE: Private Sector Development|11 000 000|
|||**4 Financing Agreements** <br>**TOTAL**|**126 000 000**|
|**Asia and Pacific**|**Asia and Pacific**|**Asia and Pacific**|**Asia and Pacific**|
|Afghanistan|DCI-ASIE/2013/024-844 (EG)|Support to credible and transparent elections|15 000 000|
|Tonga|FED/2013/024-766 (EG)|Technical Cooperation Facility, Good Governance and NSA programme II|472 000|
|Vietnam|DCI-ASIE/2013/024-370 (EG)|EU-HSPSP 2 — EU Health Sector Policy Support Programme phase 2: Towards<br>Equity and Quality of Health Services in Vietnam|114 000 000|
|Bangladesh|DCI-ASIE/2012/022-446 (EC)|Poverty Reduction Through Inclusive and Sustainable Markets (PRISM)|30 000 000|
|||**4 Financing Agreements** <br>**TOTAL**|**159 472 000**|
|||**20 Financing Agreements** <br>**TOTAL**|**459 626 000**|

## **Annexes 30**

**List of evaluations reviewed**

(euro)

|Year1|Programme<br>or Strategic<br>Evaluations|Evaluation<br>type|Programme/theme/country evaluated|Financing<br>instrument|EU<br>contribution|Evaluation<br>cost|
|---|---|---|---|---|---|---|
|2012|Strategic|Country|European Union’s cooperation with Nepal<br>— Country Level Evaluation (covered period<br>2002–10)|Mainly DCI geographic<br>and thematic and EIDHR|58 000 000|201 711|
|2011|Strategic|Thematic|European Commission support in the educa­<br>tion sector (covered period 2000–07)|DCI geographic and<br>thematic, EDF, ENPI|1 900 000 000|457 265|
|2012|Programme|Final|Food Facility — AU-IBAR projects (LEISOM<br>and VACNADA)|Food facility|2 507 860|131 870|
|2013|Programme<br>but similar to<br>Strategic|Final|Final evaluation of the whole Food Facility<br>(2009–12)|Food facility|1 000 000 000|399 720|
|2013|Programme<br>but similar to<br>Strategic|Final|Three Year Action Programme for the African<br>Peace Facility for the period 2011–13.|Mainly EDF|549 345 726|478 022|
|2012|Programme|Mid term|‘Invest in MED’|ENPI South|9 000 000|119 778|
|2013|Programme|_ex post_|‘Euromed Migration II’|ENPI South|4 994 000|148 580|
|2011|Programme|Final|Appui à l’Entretien Périodique du Réseau<br>Routier Classé|EDF|36 709 425|72 848|
|2012|Programme|Final|Programme d’appui à la décentralisation au<br>Bénin|EDF|13 785 026|47 120|
|2010|Programme|Final|Erasmus Mundus, External Cooperation<br>Window for India|DCI geographic|28 600 000|147 745|
|2014|Programme|Final|EC assisted State Partnership Programme<br>to the states of Chhattisgarh and Rajasthan<br>(budget support)2|DCI geographic|158 590 819|1 934 700|
|2011|Programme|Final|20 projects selected among 100 on EU<br>Funded Environment Projects in India|DCI geographic|340 000 000|118 008|

1 Year when the evaluation was issued.

2 The contract entails more than just an evaluation.

## **Annexes 31**

**List of ROMs reviewed**

(euro)

|Programme Title|Country/region/<br>sector|EU contribution|Report type|Report date|
|---|---|---|---|---|
|Food Facility — AU-IBAR projects (LEISOM<br>and VACNADA)|Fighting hunger|2 507 860|Ongoing|June 2011|
|FOOD FACILITY|Fighting hunger|1 000 000 000|Ongoing|May 2011|
|Public Administration Capacity Building<br>Facility Libya|Libya|4 500 000|Ongoing|October 2013|
|PROJECT ‘Euromed Migration II’|Neighbourhood South|4 994 000|Ongoing|October 2010|
|Appui à l’Entretien Périodique du Réseau<br>Routier Classé|Benin|36 709 425|Ongoing|November 2008|
|Programme d’appui à la décentralisation<br>au Bénin|Benin|13 785 026|Ongoing|February 2012|
|Programme Société Civile et Culture|Benin|14 400 000|Ongoing|May 2013|
|Sustainable Textiles for Sustainable<br>Development|India|1 672 945|Ongoing|July 2012|
|European Business and Technology Centre<br>in India (EBTC)|India|6 586 578|Ongoing|July 2012|

## **Annexes 32**

**List of delegations which replied to the survey**

|Country/Delegation|Col2|
|---|---|
|**Africa Caribbean Pacific**|**Latin America**|
|Angola|Bolivia|
|Botswana|Chile|
|Burkina Faso|El Salvador|
|Burundi|Honduras|
|Cape Verde|Nicaragua|
|Congo (Brazzaville)|Paraguay|
|Djibouti|Uruguay|
|Ethiopia|**Neighbourhood**|
|Madagascar|Algeria|
|Nigeria|Armenia|
|Rwanda|Azerbaijan|
|Togo|Belarus|
|Dominican Republic|Georgia|
|Guyana|Israel|
|Solomon Islands (Del. Papua New Guinea)|Jordan|
|**Asia**|Lebanon|
|Bangladesh|Morocco|
|Cambodia|Tunisia|
|India|Ukraine|
|Laos, Thailand||
|Nepal|Nepal|
|Pakistan|Pakistan|
|Philippines|Philippines|
|Vietnam|Vietnam|

#### **Reply of the** **Commission**

**Executive summary**

**II**
The Commission considers that the systems for
strategic evaluations are overall reliable even if they
could be improved. For programme evaluations
and ROM, systems exist but in some areas are not
functioning properly.

**III**
The systems exist for supervision of programme
evaluations, and steps are being taken to make
them more effective.

The Commission considers that financial allocations
are well managed for strategic evaluations and ROM
and that improvements are possible for programme
evaluations.

The human resources management for evaluation
and ROM should be considered in the framework of
the overall human resources management for the
external assistance managed by EuropeAid.

**IV**
The Commission’s evaluation findings are based on
internationally recognised methodology agreed in
the OECD. EuropeAid’s evaluation plans are drawn
up to respond to the aims of learning and accounta­
bility, are implemented in accordance with common
principles of evaluation and are monitored through
regular internal reports.

Guidance is going to be provided to staff on
drawing up plans for programme evaluations and
monitoring activities, ensuring complementarity
between them. The selection of projects to be
subject of a ROM will, as from January 2015, take
into account better, whether or not an evaluation is
foreseen.

## **33**

A system for monitoring the implementation of
programme evaluation plans already exists in the
EAMR, and progress in strategic evaluations is
reported to Commissioners.

The monitoring system for programme evaluations
will be improved and more emphasis will be put on
quality control issues for the programme evalua­
tions. Improved guidance and assistance will also
be offered to the project managers through the
information management system for programme
evaluations which is under development.

With respect to ROM, the Commission has initiated
reforms that should substantially improve the qual­
ity of the system, including in particular external
quality assurance.

**V**
Mechanisms for follow-up and dissemination are
already in place. For ROM, the existing mechanism
is being improved. Follow-up to strategic evalua­
tions is formalised and published.

For programme evaluations, the Commission recog­
nises that improvements are possible with respect
to follow-up, oversight and monitoring, functionali­
ties that will be covered by the information man­
agement system under construction, providing
moreover appropriate access to programme evalua­
tions, and ensuring better dissemination.

**VI**
The smart objectives and verifiable indicators are
defined at the level of multiannual programming
and project documents.

The choice on whether to carry out an _ex post_ evalu­
ation is linked to criteria of efficiency and relevance.
The number of individual _ex post_ evaluations as
such has no relevance.

## **34**

**Reply of the Commission**

The Commission discontinued _ex post_ ROM under
the revised ROM system as it is not a monitoring
tool and as it is moreover not considered to be an
appropriate tool to assess impact.

The Commission considers that there are no inher­
ent limitations in the evaluation methodology for
budget support.

**Introduction**

**03**
The latest (fourth) Article 318 Report was adopted
on 26 June 2014. It builds on the experience gath­
ered with the previous editions of the report and
takes due account of the recommendations made
by the Court of Auditors and the European Parlia­
ment, including those made in the EP resolution of
26 February 2014.

**09**
The main system used by Europe-Aid to moni­
tor programme implementation on the spot is
the internal monitoring by the Commission’s own
services and by EU delegations in accordance with
the 2007 guidelines on internal monitoring and the
2012 budget support guidelines. ROM has therefore
been an additional system of programme perfor­
mance monitoring implemented by external con­
sultants with respect to programmes implemented
through the project modality.

**Observations**

**The human resources management for evaluation**
**and ROM should be considered in the framework**
**of the overall human resources management for**
**the external assistance managed by EuropeAid.**

**17**
The delegations and operational units already
report to EuropeAid Management via the annual
evaluation plan in the Annual Management Plan,
and report systematically to headquarters on the
implementation of their evaluation plan in their
EAMR.

Instructions and guidance are under preparation
to staff on the preparation and implementation of
their future monitoring and evaluation plans and
to the follow-up to be given to the conclusions
and recommendations of evaluations. Moreover an
information management system is under devel­
opment that will facilitate the management and
monitoring of programme evaluations. The network
of Evaluation Correspondents in delegations and
operational units, set up in 2013, will provide added
feedback.

**The human resources management for evaluation**
**and ROM should be considered in the framework of**
**the overall human resources management for the**
**external assistance managed by EuropeAid.**

**19**
The information management system under con­
struction will provide more precise information on
the number of programme evaluations and their

cost.

**20**
Monitoring, reporting, mid-term evaluation and,
more generally, project management activities are
so closely interlinked that any division of resource
allocation between them can only be an estimate.
This is even more so with respect to the new ROM
which is to be a tool for support to overall monitor­
ing of programmes.

**21**
The human resources management for evaluation
and ROM should be considered in the framework of
the overall human resources management for the
external assistance managed by EuropeAid.

**22**
Avoiding ROM when an evaluation is to take place
was already a principle before the ongoing reform
of monitoring and evaluation systems (as men­
tioned also by the Court). This reform also includes
the introduction of criteria for the selection and
planning of programme evaluations. Furthermore,
a new requirement for delegations and opera­
tional units to draw up an annual monitoring and

## **35**

**Reply of the Commission**

evaluation plan will be introduced, ensuring com­
plementarity between ROM reviews and mid-term
programme evaluations.

_**The Commission considers that EuropeAid’s evalua­**_
_**tion and ROM systems provide relevant and robust**_
_**findings, even if improvements are needed.**_

**26**
Instructions and guidance to staff on the selection
criteria for programme evaluations and the plan­
ning of evaluations within the framework of the
preparation of monitoring and evaluation plans will
be part of the overall instructions and guidance on
the reforms of monitoring, reporting and evalua­
tion. This guidance will be issued before the end
of 2014. Selection criteria are to be reflected in the
financing agreements.

**26 (a)**
Mid-term evaluations are made compulsory on
a case-by-case basis at formulation phase taking
into account the programme specificity.

**26 (b)**
_Ex post_ evaluations are made compulsory on a caseby-case basis at formulation phase taking into
account the programme specificity.

_**Evaluations are delivered as planned in a majority**_
_**of cases.**_

**27**
Systems are in place but the Commission acknowl­
edges the need to improve them.

**28**
In addition to staff constraints, fewer strategic
evaluations than planned were completed because
of the political situation in various countries, the
increasing complexity of evaluations (particularly
evaluations carried out jointly with partners and
other donors), and also to changes in the contrac­
tual framework for these evaluations.

**29**
The Commission reminds that the implementation
rate is above 80 %, which is the EuropeAid's bench­
mark (KPI 19 EAMR).

The EAMR will have to provide improved documen­
tation of the non-compliance of the planning of
the programme evaluations that will be established
by Delegations and operational HQ services in the
upcoming Monitoring and evaluation plans.

**31**
The revised new ROM system as adopted in May
2014 as part of a wider reform of monitoring and
reporting has the primary objective of improving
the internal monitoring and reporting on perfor­
mance and results by delegations and operational
units and of reducing a too heavy reliance on
external ROM, in line with the Court's special report
No 1/2011 on devolution. The main objective of
the new ROM system is to provide support to the
internal monitoring and reporting. The selection
methodology reflects these priorities:

— For internal monitoring purposes, the selection

methodology for ROM review focuses essen­
tially on projects and programmes flagged by
delegations and operational units as (1) facing
problems, (2) for which sector expertise is lack­
ing and (3) that could not be visited.

— For reporting on results purposes, the system

is to support end of projects results reporting,
covering all projects and programmes above
EUR 750 000 ending in a given year and to assist
delegations and operational units to report on
results, including for the aggregated results on
the basis of the Commission’s future EU Devel­
opment and Cooperation Results framework.

The Commission therefore considers that by includ­
ing a risk-based selection for ongoing programmes,
while ending the use of ROM as an indicator for the
overall performance measurement of its portfolio,
the new system will enhance the usefulness of ROM,
in accordance with its main objective. While ROM
is not designed for the purpose of measuring the
overall performance of the Commission’s aid portfo­
lio, the strengthening of the internal monitoring

## **36**

**Reply of the Commission**

and reporting on performance and results and the
planned reporting on aggregated results on the
basis of the Commission’s future EU Development
and Cooperation Results framework are to be part
of the tools that should allow the Commission
to better measure the overall performance of its
portfolio.

**33**
Under the new ROM contracts to be signed in 2014,
all the reviews and other support missions are to
be implemented by senior experts who should
moreover be sector experts. Besides, a significant
number of reviews should indeed be carried out by
key experts on a full time assignment whereas for
the other reviews the experts will be hired on an ad
hoc basis.

**36**
EuropeAid has emphasised in its recent Evalua­
tion Policy the importance of adhering to existing
guidance on setting up reference groups. (See for
existing guidelines: http://ec.europa.eu/europeaid/
evaluation/methodology/guidelines/gba_det_
en.htm#01_03 and http://ec.europa.eu/europeaid/
evaluation/methodology/examples/131011-tor-pro­
ject-evaluations-guidance_en.pdf),

**37**

The introduction in 2013 of the Evaluation Corre­
spondents’ network aims to strengthen significantly
the evaluation function across the system, including
in Delegations.

More emphasis will be put on quality control issues
for the programme evaluations.

**38**

Several measures to be taken under the revised
ROM system should solve the deficiencies charac­
terising the present situation: the strengthening of
the quality of ROM reviews, the improved condi­
tions for appropriate internal quality control by the
ROM contractors, the setting up of external quality
assurance, the importance to be given to defining
the follow-up, and improved overall monitoring of
implementation of the revised ROM system. The
encoding of the follow-up planned to take place in
the new information management module for ROM
is another measure that should contribute to the

same.

**The Commission considers that the evaluation and**
**ROM systems provide adequate information on**
**results achieved, even if improvements are needed.**

_**For the new programming period 2014–20, the Com­**_
_**mission has stepped up its efforts to have smart objec­**_
_**tives and verifiable indicators. The new programming**_
_**documents have been reviewed in that sense and the**_
_**operationalisation of those efforts at project level has**_
_**been initiated.**_

**40**
Objectives and indicators for regional and world­
wide thematic programmes have to be general in
order to capture a variety of situations. (see reply to
Box 1).

**Box 1**
Investing in people is a worldwide thematic pro­
gramme covering all developing countries. This
implies that the objectives are in many cases
rather high level precisely to cover many different
situations.

The objective quoted for the 2007–13 Bolivia
programming document represents rather an
overall (or general) objective than a set of specific
objectives.

## **37**

**Reply of the Commission**

**41**
In programmes covering several countries and
regions, the objectives of the programme give the
key priorities; the specific objectives for each subprogramme are more precise and indicators should
be measurable.

**Box 2**
The Commission wants to highlight that these
examples refer to general objectives.

Concerning the Euromed migration programme,
the objectives are only partially quoted by the
Court. The full objectives can be found at the fol­
lowing address: http://www.enpi-info.eu/mainmed.
php?id_type=10&id=9

**42**
The purpose of the study to which the Court refers
was to review the evaluations related to the legal
instruments for the last programming period (2007–
13). The findings of the study have been taken into
account to prepare the instruments covering the
MFF 2014–20.

_**Strategic evaluations focus on the chain of**_
_**results achieved and not on implementation of**_

_**programmes.**_

**Common Commission reply to**
**paragraphs 43 to 45**
The decision to undertake an _ex post_ evaluation
of a programme is a question of efficiency and
relevance. The decision should not be taken in
isolation but with reference to the financial weight
and strategic importance of the programme to be
evaluated.

**46**
Under the revised ROM system, the Commission
discontinued _ex post_ ROM as it is not a monitoring
tool and is not considered to be an appropriate tool
to assess impact.

**47**
The approach of the EIB referred to by the Court
is specific and is to be seen in connection with
the nature of the loan-based operations managed
by the EIB. It cannot be compared to the case of
a grant donor like the EU for the grant based fund­
ing of programmes.

The ongoing set-up of the EU Development and
Cooperation Results Framework through which the
Commission aims to implement a first reporting in
2015 does not proceed on such an approach.

_**Methodology to evaluate budget support has**_
_**proved its strong results and has been recognised at**_
_**international level.**_

**48**
The document setting out the methodological
approach to evaluating budget support developed
under the auspices of the OECD-DAC, provides
a robust approach addressing the complexity of
evaluating budget support.

**49**
Robustness of the approach is recognised by the
international evaluation community, and it has
been published on the OECD website.

Consequently, the Commission does not see any
reason to mention any limitations in its methodol­
ogy. So far, seven evaluations of budget support
have been finalised which have been able to assess
the results of budget support and have made rec­
ommendations for improvement.

_**Evaluations are a tool for assessment of the whole**_
_**chain of results, but as regards outputs and direct**_
_**outcomes internal monitoring provides primary**_
_**information and is used to ensure reporting on**_
_**results across the portfolio.**_

**50**

The Commission considers that ROM is not the
appropriate tool to provide information on sus­
tainability and impact and that _ex post_ evaluations
should be only decided on a case-by-case basis.

## **38**

**Reply of the Commission**

**52**
When preparing the latest (fourth) edition of
the Article 318 Evaluation Report, adopted on
26 June 2014, the Commission has duly taken into
account the observations made by the Court of
Auditors in its 2012 annual report.

The Commission also refers to its replies to the
Court’s observations under Chapter 10 of the ECA
2012 Annual Report where it provides its position
with regard to the performance information in the
third and future editions of the Art. 318 evaluation

report.

**Follow-up mechanisms are in place, although they**
**could be improved.**

**54**
Given the complexity of strategic evaluations, it is
difficult to have clear cut commitments linked to
high level recommendations. In 2013, the Evalua­
tion unit carried out a study on uptake of strategic
evaluations results. On that basis, the management
response system is being reviewed. In any case, the
responsibility lies clearly with the geographic or
thematic services responsible for the implementa­
tion of the programmes or policies.

**55**
The review of the management response system
will take into account not only this issue of the dura­
tion of the follow-up period but also other issues
like brokering the evaluation results and involve­
ment of management.

**56**

The Evaluation unit does know which recommen­
dation has been accepted. The completed ‘fiches
contradictoires’ indicate clearly whether a given
recommendation is accepted, partially accepted
or not accepted. On the basis of the study referred
to by the Court, the Commission has adopted an
action plan to improve the uptake of strategic
evaluation results.

_**The Commission will take additional measures to**_
_**improve the documentation of follow-up for pro­**_
_**gramme evaluations and ROM.**_

**57**
Follow-up to programme evaluations is part of the
the lessons learnt section of new project propos­
als (identification and formulation stages of new
projects) when deemed relevant.

The guidance note accompanying the updated
terms of reference template for programme evalu­
ations issued in 2013 emphasised the need to
document the management response. For ROM, the
revised system is to include appropriate measures
to ensure the definition of an appropriate followup plan to a ROM review and related monitoring of
such follow-up.

For programme evaluations, the point will be taken
into account in the upcoming instruction guidance
as well as in the information management system
under construction.

**58**
For ROM, as stated in the reply to §59, the revised
system is to include appropriate measures to ensure
the definition of an appropriate follow-up plan
to a ROM review and related monitoring of such
follow-up.

**The Commission is in the process of develop­**
**ing an information management system needed**
**for improving dissemination of programme**
**evaluations.**

**60**
The aggregation of ROM performance data in its
previous form in AAR and AR will be discontinued.

## **39**

**Reply of the Commission**

_**The Commission is in the process of developing the**_
_**related information management system needed**_
_**for improving such dissemination.**_

**61**
Dissemination of programme evaluations will be
improved:

— Firstly, by improved access to such evaluations

through the setting up of an information man­
agement system;

— Secondly, on that base by improved analysis of

such evaluations by thematic units; and

— Finally, by better contacts between Delegations

and in-country stakeholders.

The Evaluation Correspondents’ Network can play
an important part in this.

**62**
The information management system that is to con­
tain the database for programme evaluations is also
to contain the lessons learned.

**63**
The Commission is in the process of developing
such an information management system needed
for improving such dissemination.

**64**
The IT tool covering both ROM and programme
evaluations, was already under construction at the
time of the audit.

This development has, in 2014, undergone a delay
as a result of the overall revision of the direction to
take with respect to EuropeAid’s new operational
information management system that is to be
developed. This does however not put into ques­
tion the need to develop such a system.

**Conclusions and recommendations**

**65**
The Commission considers that the systems for
strategic evaluations are overall reliable even if they
could be improved. For programme evaluations
and ROM, systems exist but in some areas are not
functioning properly.

**66**
The systems exist for supervision and use of pro­
gramme evaluations, and steps are being taken to
make them more effective.

The Commission considers that financial allocations
are well managed for strategic evaluations and ROM
and that improvements are possible for programme
evaluations.

The human resources management for evaluation
and ROM should be considered in the framework of
the overall human resources management for the
external assistance managed by EuropeAid.

**67**
The Commission’s evaluation findings are based on
internationally recognised methodology agreed in
the OECD. Guidance is going to be provided to staff
on drawing up plans for programme evaluations
and monitoring activities, ensuring complementa­
rity between them.

A system for monitoring the implementation of
programme evaluation plans already exists in the
EAMR, and progress in strategic evaluations is
reported to Commissioners.

With respect to ROM, the Commission has initiated
reforms that should substantially improve the qual­
ity of the system, including in particular external
quality assurance.

## **40**

**Reply of the Commission**

**68**
Mechanisms for follow-up and dissemination are
already in place. For ROM, the existing mechanism
is being improved. Follow-up to strategic evalua­
tions is formalised and published.

For programme evaluations, the Commission recog­
nises that improvements are possible with respect
to follow-up, oversight and monitoring, functionali­
ties that will be covered by the information man­
agement system under construction, providing
moreover appropriate access to programme evalua­
tions, and ensuring better dissemination.

**69**
The smart objectives and verifiable indicators are
defined at the level of multiannual programming
and project documents.

The choice on whether to carry out an _ex post_ evalu­
ation is linked to criteria of efficiency and relevance.
The number of individual _ex post_ evaluations as
such has no relevance.

The Commission discontinued _ex post_ ROM under
the revised ROM system as it is not a monitoring
tool and as it is moreover not considered to be an
appropriate tool to assess impact.

The Commission considers that there are no inher­
ent limitations in the evaluation methodology for
budget support.

**70**
The revised ROM reform adopted by EuropeAid in
May 2014 aims not only to clarify the distinctions
between ROM and evaluation but also to make the
system more efficient; this will be made operational
in the detailed instructions to be sent to staff by the
end of 2014. Following publication of the evalua­
tion policy document ‘Evaluation matters’, reflect­
ing a major process of consultation and discussion
between October 2013 and March 2014, the guid­
ance to staff on the principles, responsibilities and
procedures for evaluations will be updated and
reinforced.

**Recommendation 1**
The Commission accepts the recommendation.

The human resources management for evaluation
and ROM should be considered in the framework of
the overall human resources management for the
external assistance managed by EuropeAid.

**Recommendation 2**
— The Commission accepts this recommendation.

Guidance is to be provided to staff on drawing
up plans for programme evaluations and moni­
toring activities, ensuring complementarity
between them. The selection of projects to be
subject of a ROM will, as from January 2015, bet­
ter take into account whether or not an evalua­

tion is foreseen.

— The Commission accepts this recommendation.

For programme evaluation plans, it will be done
on the basis of the EAMR.

— The Commission accepts this recommendation.

**Recommendation 3**
— The Commission accepts this recommendation.

More emphasis will be put on quality control is­
sues for the programme evaluations. Improved
guidance and assistance will also be offered to
the project managers through the information
management system for programme evalua­
tions which is under development.

— The Commission accepts this recommendation.

The monitoring system for programme evalua­
tions will be improved.

## **41**

**Reply of the Commission**

**Recommendation 4**
— The Commission accepts this recommendation

and promotes the use of SMART objectives and
verifiable indicators.

— The Commission does not accept this recom­

mendation as the cost benefit is not proven.

— The Commission accepts this recommendation,

if further analysis shows an increase in _ex post_
evaluations is efficient and useful.

**Recommendation 5**
— The Commission partially accepts this recom­

mendation. Extension of the follow-up period is
one of the options being considered in a review
of the follow-up system.

— The Commission accepts this recommendation.

Such a database is already under construction.

**HOW TO OBTAIN EU PUBLICATIONS**

**Free publications:**

- one copy:
via EU Bookshop (http://bookshop.europa.eu);

- more than one copy or posters/maps:
from the European Union’s representations (http://ec.europa.eu/represent_en.htm);
from the delegations in non‑EU countries (http://eeas.europa.eu/delegations/index_en.htm);
by contacting the Europe Direct service (http://europa.eu/europedirect/index_en.htm) or
calling 00 800 6 7 8 9 10 11 (freephone number from anywhere in the EU) (*).

(*) The information given is free, as are most calls (though some operators, phone boxes or hotels may charge you).

**Priced publications:**

- via EU Bookshop (http://bookshop.europa.eu).

**Priced subscriptions:**

- via one of the sales agents of the Publications Office of the European Union
(http://publications.europa.eu/others/agents/index_en.htm).nts/index_en.htm).