Source: EURLEX
Language: en
Format: md

|  |  |  |  |
| --- | --- | --- | --- |
| 30.1.2008 | EN | Official Journal of the European Union | C 26/1 |

---

SPECIAL REPORT No 9/2007

concerning ‘Evaluating the EU Research and Technological Development (RTD) framework programmes — could the Commission's approach be improved?’ together with the Commission's replies

(pursuant to Article 248(4), second subparagraph, EC)

(2008/C 26/01)

TABLE OF CONTENTS

|  |  |
| --- | --- |
| I-V | EXECUTIVE SUMMARY |

|  |  |
| --- | --- |
| 1-14 | INTRODUCTION |

|  |  |
| --- | --- |
| 1-7 | The role of the European Union in supporting RTD activities |

|  |  |
| --- | --- |
| 8-14 | Evaluation and monitoring of FPs |

|  |  |
| --- | --- |
| 15-18 | AUDIT SCOPE AND APPROACH |

|  |  |
| --- | --- |
| 19-109 | AUDIT OBSERVATIONS |

|  |  |
| --- | --- |
| 19-37 | Intervention logic, objective setting and performance indicators |

|  |  |
| --- | --- |
| 38-49 | Evaluation strategy |

|  |  |
| --- | --- |
| 50-68 | Organising evaluation for the FPs |

|  |  |
| --- | --- |
| 69-97 | Evaluation methodologies and techniques |

|  |  |
| --- | --- |
| 98-109 | Disseminating and using evaluation findings |

|  |  |
| --- | --- |
| 110-118 | OVERALL CONCLUSION |

Annex I: Measuring the impact and results of public intervention in RTD

Annex II: Provisions regarding monitoring and evaluation in legislative decisions for FP4 to FP7

Annex III: Main differences the in ex-post evaluation of research programmes between the EU (FP4 — FP6), USA and Canada

Annex IV: Possible panel structure for FP7

The Commission's replies

EXECUTIVE SUMMARY

|  |  |
| --- | --- |
| I. | The Community RTD framework programmes (FPs) are the largest public source of research funding in the European Union. The Commission is responsible for managing the FPs and assessing their results. |

|  |  |
| --- | --- |
| II. | The audit covered the monitoring and evaluation arrangements in place since 1995 for the last three programming periods (FP4, FP5, FP6) and also gives an outlook for FP7 (2007-2013). The audit was carried out with the EFTA Board of Auditors and the assistance of external experts. |

|  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| III. | The audit addressed the question of whether the Commission's approach to assessing the results of the FPs was adequate. In this context, the Court checked whether the Commission met the legal requirements for evaluation and ascertained whether its system for evaluation and monitoring met stakeholder expectations. In particular, the Court asked whether:   |  |  | | --- | --- | | (i) | there was an underlying intervention logic to the FPs and their constituent programmes that could provide a sound basis for evaluation and monitoring; |  |  |  | | --- | --- | | (ii) | the Commission had defined an evaluation strategy for the FPs; |  |  |  | | --- | --- | | (iii) | the organisational structure of the Commission's evaluation system for the FPs ensured that evaluation was properly coordinated, adequately resourced and perceived by stakeholders as being sufficiently independent; |  |  |  | | --- | --- | | (iv) | methodologies used for evaluation were satisfactory; and |  |  |  | | --- | --- | | (v) | evaluation and monitoring activities provided information and analysis which could be used to improve ongoing programmes and prepare future FPs. | |

|  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| IV. | The Court's findings and conclusions were as follows:   |  |  | | --- | --- | | (i) | the lack of an explicit intervention logic and the presence of poorly defined programme objectives and weak performance measurement undermined effective monitoring and evaluation; |  |  |  | | --- | --- | | (ii) | the absence of a comprehensive evaluation strategy, agreed among the ‘research DGs’ implementing the FPs, resulted in inconsistent approaches between the different Commission services. In particular, this is illustrated by an under-evaluation of significant parts of the FPs; |  |  |  | | --- | --- | | (iii) | evaluation of the FPs was decentralised, the existing coordination mechanisms among the DGs implementing the FPs were not effective and the Commission's central services had no enforcement role. In general terms, however, the ‘research DGs’ have complied with the Commission's formal requirements on evaluation. The use of panels composed of high-level external experts appointed by the Commission for major evaluation exercises was well suited to the Community context. However, these temporary panels were not involved early enough in the process to influence evidence collection and had only limited possibilities to follow up relevant issues over time. By using only an overall FP panel (in addition to a panel for one specific programme), insufficient attention was given to specific issues related to the different scientific areas within the FPs; |  |  |  | | --- | --- | | (iv) | inadequate methodological guidance was provided, evaluators found difficulties in gathering relevant data and there were no evaluation studies that addressed the longer-term outcomes and impacts of the FPs, as opposed to short-term issues of programme implementation; and |  |  |  | | --- | --- | | (v) | as a result of the above, and because the timing of evaluations was often premature for addressing the most relevant issues, the Commission's evaluation system for the FPs was of limited usefulness to policy-makers, stakeholders or even the Commission itself. Ultimately, little is known about the achievement of programme objectives and the results of the FPs. | |

|  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| V. | The Court recommends that:   |  |  | | --- | --- | | (i) | intervention logic should be rendered explicit in future legislation. Underlying assumptions should be explained, the link between scientific and socio-economic objectives clarified and appropriate performance indicators developed; |  |  |  | | --- | --- | | (ii) | a comprehensive evaluation strategy should be developed by (and agreed among) the DGs implementing the FPs. In particular, this should entail a consistent approach with regard to the minimum level at which detailed evaluation must take place so as to take account of the specificities of each scientific field; |  |  |  | | --- | --- | | (iii) | consideration should be given to setting up a joint evaluation office for coordinating the ‘research DGs’ evaluation activities for the FP as a whole and creating a system of panels (and sub-panels) composed of external experts. These panels should be set up sufficiently early in order to provide effective assistance to the Commission, and continue to do so throughout the programming period and thereafter; |  |  |  | | --- | --- | | (iv) | the data requirements for evaluation and monitoring should be analysed properly and more extensive use should be made of other existing sources of data. The Commission should also rationalise the reporting requirements for participants. Finally, it should draw up a comprehensive evaluation manual for the FPs and develop a broader range of evaluation methodologies and techniques to be used for evaluations in this field; and |  |  |  | | --- | --- | | (v) | the Commission should establish the type and scope of evaluation that can be reasonably expected for the dates specified in the FP7 legislation and clarify how evaluations can be used to adapt programmes (learning programmes) and what contribution they can make to policy decisions. | |

INTRODUCTION

The role of the European Union in supporting RTD activities

|  |  |
| --- | --- |
| 1. | Research and technological development (RTD)[(1)](#ntr1-C_2008026EN.01000101-E0001) is the main driver of scientific and technological progress and innovation and, as such, is an important contributor to improvements in living standards. Public intervention in RTD is generally justified as a corrective to so-called ‘market failures’, with public support reducing uncertainty, providing investment where research is a public good or where there are significant externalities. |

|  |  |
| --- | --- |
| 2. | Article 163 of the EC Treaty establishes the Communities' role in supporting RTD, stipulating that ‘the Community shall have the objective of strengthening the scientific and technological bases of Community industry and encouraging it to become more competitive at international level, while promoting all the research activities deemed necessary …’. Articles 164-173 determine the activities to be carried out in this respect and the scope and implementation of the multi-annual RTD framework programmes (FPs). The Commission is responsible for putting forward proposals for the FPs, whereas the FP decisions and the rules for participation and dissemination are adopted by co-decision of the European Parliament and the Council. The latter adopts the specific programmes alone. The FPs are implemented jointly by six Directorates-General (DGs), the so-called ‘research DGs’[(2)](#ntr2-C_2008026EN.01000101-E0002). |

|  |  |
| --- | --- |
| 3. | Within the wide range of policies implemented in the European Union to strengthen innovation and competitiveness, the FPs are the most important financial instrument contributing to the Lisbon strategy[(3)](#ntr3-C_2008026EN.01000101-E0003) and the Barcelona objective[(4)](#ntr4-C_2008026EN.01000101-E0004) at the Community level.  Image |

|  |  |
| --- | --- |
| 4. | Through the FPs, the Community provides funding to researchers within the European Union, associated countries and international organisations. Their budgets have increased significantly over the years, reaching 7 217 million euro per year under FP7 as against 2 761 million euro under FP4 (see Figure 1). In terms of budgetary appropriations, they represent the largest area of direct centralised management within the Commission[(5)](#ntr5-C_2008026EN.01000101-E0005). |

|  |  |
| --- | --- |
| 5. | The FPs differ from many national programmes in that they cover both basic and applied research with the participation of industry and public research organisations[(6)](#ntr6-C_2008026EN.01000101-E0006). They are made up of specific programmes (and sub-programmes) dealing with broad scientific fields. These programmes typically consist of a number of domains that concentrate on more specific scientific areas. These domains are then implemented through projects (called indirect actions) following calls for proposals. In this way, hundreds of individual projects are funded per domain. |

|  |  |
| --- | --- |
| 6. | The overall share of the FPs in total public RTD funding within the European Union and its Member States ranges between 4 % and 5 %. Nevertheless, its impact on which kind of research activities are carried out within Europe is significantly larger. This is because Community grants generally provide only part of the total funding for a project. In addition, when applying for funding, researchers need to demonstrate that their projects address the objectives specified in the calls for proposals. In this way, the FPs set incentives to orient RTD activities towards specific EU objectives. |

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 7. | In recent years significant changes have been made to the FPs, due to:   |  |  | | --- | --- | | — | new political challenges (the Lisbon strategy and Barcelona objectives, the establishment of a ‘European Research Area’ (ERA) and the increasing use of the open method of coordination[(7)](#ntr7-C_2008026EN.01000101-E0007) in research and innovation policy), |  |  |  | | --- | --- | | — | the changing orientation of FPs, from simply solving technical problems under FP4 and addressing specific social problems under FP5, to encouraging the co-operation of research organisations within the ERA and contributing to the establishment of a knowledge-based economy and society under FP6 and FP7, |  |  |  | | --- | --- | | — | longer programming periods (previously 4 years, now 7 years for FP7 in line with the 2007-2013 Financial Perspectives), |  |  |  | | --- | --- | | — | more Member States and participating countries, |  |  |  | | --- | --- | | — | increased funding, in particular under the current Financial Perspectives, |  |  |  | | --- | --- | | — | the setting up of the European Research Council (ERC) and the delegation of programme management activities to executive agencies starting under FP7, and |  |  |  | | --- | --- | | — | the increasing use of large-scale funding instruments (e.g. Networks of Excellence, Integrated Projects, Article 169 projects, Joint Technology Initiatives). | |

Evaluation and monitoring of FPs

|  |  |
| --- | --- |
| 8. | The Commission has responsibility for programme implementation and the assessment of programme results[(8)](#ntr8-C_2008026EN.01000101-E0008). Within the Commission, evaluation is decentralised to the DGs, with the central services (in particular DG Budget and the Secretariat-General) providing support and coordination. |

|  |  |
| --- | --- |
| 9. | This report employs the Commission's own definitions of ‘evaluation’ as a ‘judgement of interventions according to their results, impacts and needs they aim to satisfy’[(9)](#ntr9-C_2008026EN.01000101-E0009), and ‘monitoring’ as providing information on the implementation of ongoing programmes and projects to address management needs[(10)](#ntr10-C_2008026EN.01000101-E0010). This is also how the two activities are distinguished in the current FP7 legislation. |

|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 10. | Hereinafter, the following terminology is used to describe the results of RTD activities:   |  |  | | --- | --- | | — | ‘outputs’ are knowledge, goods and services directly produced as a consequence of the public intervention (publications, conference papers, patents, licences, prototypes, standards, trained scientists, etc.), |  |  |  | | --- | --- | | — | ‘outcomes’ are the delayed result of the users' processing of outputs, and |  |  |  | | --- | --- | | — | ‘impacts’ are the long-term socio-economic and political changes the public intervention brings about. |   Box 1Example of the ‘Global System for Mobile communications’ (GSM)In this example, the ‘outputs’ of Community funding were the immediate project results, such as the development of the GSM standard. The ‘outcome’ was its acceptance as the digital cellular telephony standard within Europe and beyond, which contributed greatly towards the development of the telecommunication industry. Finally, ‘impacts’ were the economic, social and political consequences of the international use of this technology. This includes contributing towards job creation within the telecommunications and related industries, and the growth of the importance of this industrial sector for the European economy. GSM has also ultimately provided a new way for people to communicate with each other. |

|  |  |
| --- | --- |
| 11. | Assessing those results, however, poses a number of methodological difficulties (see Annex I). Socio-economic changes are complex and, more specifically, the relationship between a research activity, and the outcomes and impacts that are observed is often difficult to trace. While representing a considerable challenge, these methodological difficulties do however not necessarily preclude effective evaluation. |

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 12. | The Commission has had a system to monitor and evaluate the FPs in place since the 1980s[(11)](#ntr11-C_2008026EN.01000101-E0011). Starting with FP4 in the mid-1990s, the monitoring and evaluation system consisted of a number of partly linked activities, including:   |  |  | | --- | --- | | — | annual monitoring (mainly of programme implementation), and |  |  |  | | --- | --- | | — | five-year assessments (5YAs) of FP activities (and, as from FP6, additional assessments of specific issues). |   Both activities had to be carried out with the assistance of external experts (see Figure 2 and Annex II).  Figure 2  FP4 to FP7 — key requirements for monitoring and evaluation   |  |  |  |  | | --- | --- | --- | --- | | FP4 (1995-1998) | FP5 (1999-2002) | FP6 (2003-2006) | FP7 (2007-2013) | | two evaluation exercises (mid-term and final), in each case for its 20 sub-programmes plus the FP; final evaluation overlaps with mid-term of FP5 | two simultaneous evaluation exercises (mid-term), in each case for its 7 Specific Programmes plus the FP | one evaluation exercise for the FP and its 3 Specific Programmes (mid-term), but not specifically for all 20 sub-programmes, complemented by specific evaluation and regular progress (i.e. monitoring) reports | at least five evaluation exercises (ex-ante, mid-term and final) including specific progress report before interim evaluation | | |  |  | | --- | --- | | 1. | Ex-ante: no | | |  |  | | --- | --- | | 1. | Ex-ante: no | | |  |  | | --- | --- | | 1. | Ex-ante: no | | |  |  | | --- | --- | | 1. | Ex-ante: incorporated in ‘Impact Assessment of the Commission's proposal for FP7’ (2005) | | | |  |  |  |  | | --- | --- | --- | --- | | 2. | Mid-term:  |  |  | | --- | --- | | — | 5YA: mid-way through FP, prior to subsequent FP (1997) | | | |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | 2. | Mid-term:  |  |  | | --- | --- | | — | 5YA: prior to presenting proposal for subsequent FP (2000) |  |  |  | | --- | --- | | — | progress review mid-way trough FP (2001) | | | |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | | 2. | Mid-term:  |  |  | | --- | --- | | — | evaluation of effectiveness of instruments (2004) |  |  |  | | --- | --- | | — | 5YA, prior to presenting proposal for subsequent FP (2004) |  |  |  | | --- | --- | | — | regular progress (i.e. monitoring) reports | | | |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | | 2. | Mid-term:  |  |  | | --- | --- | | — | ex-post, two years after the end of previous FP (2008) |  |  |  | | --- | --- | | — | progress report before interim evaluation (before 2010) |  |  |  | | --- | --- | | — | interim evaluation (2010) | | | | |  |  |  |  | | --- | --- | --- | --- | | 3. | Final:  |  |  | | --- | --- | | — | on completion on FP (1999) | | | |  |  | | --- | --- | | 3. | Final: no | | |  |  | | --- | --- | | 3. | Final: no  (modified by to FP7 Decision) | | |  |  | | --- | --- | | 3. | Final: two years after completion of FP (2015), supported by specific studies, the interim evaluation and other evaluation activities carried out throughout the FP period | | | Source: see Annex II. | | | | |

|  |  |
| --- | --- |
| 13. | To date, three 5YAs have been carried out[(12)](#ntr12-C_2008026EN.01000101-E0012), leading to the ‘Davignon’, ‘Majó’ and ‘Ormala’ reports (1997, 2000 and 2004 respectively). In addition, DG Information Society carried out targeted assessments for its specific programme. It should be noted that 5YAs were discontinued under FP7 legislation, thereby removing the constraint that major evaluation exercises had to concentrate on the preceding five years and, as a result, had to cover parts of two successive programming periods. In 2004, the ‘Marimón’ report[(13)](#ntr13-C_2008026EN.01000101-E0013), a specific evaluation regarding the effectiveness of the new instruments introduced under FP6, was also published. |

|  |  |
| --- | --- |
| 14. | The Commission's framework for evaluation activities was reviewed in 2000 and 2002[(14)](#ntr14-C_2008026EN.01000101-E0014). These communications were updated in 2007 taking account of the consequences of the Commission's administrative reform and the new Financial Regulation[(15)](#ntr15-C_2008026EN.01000101-E0015). In these documents, the role and function of evaluation is described in the context of the Commission's overall planning, programming and reporting cycle and a set of 21 evaluation standards and 18 models of ‘good practice’ are specified. Furthermore, these documents discuss the use of evaluation in the preparation and simplification of EU legislation. |

AUDIT SCOPE AND APPROACH

|  |  |
| --- | --- |
| 15. | The main objective of the audit was to determine whether the Commission had an adequate approach to assessing the results of the RTD framework programmes. |

|  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 16. | To this end, the Court assessed the extent to which:   |  |  | | --- | --- | | — | an explicit intervention logic underlies the FPs and their constituent programmes around which outputs, outcomes and impacts can be measured and assessed, thereby providing a sound basis for evaluation and monitoring, |  |  |  | | --- | --- | | — | an evaluation strategy has been defined for the FPs promoting effective and consistent evaluation, |  |  |  | | --- | --- | | — | the organisational structure of the evaluation system ensures that it is appropriately coordinated across the FP, adequately resourced and also perceived by stakeholders as being sufficiently independent, |  |  |  | | --- | --- | | — | the methodologies used for evaluation ensure that relevant issues are addressed in an appropriate way, so that evaluation is perceived as credible, and |  |  |  | | --- | --- | | — | evaluations and monitoring activities provide information and analysis which can be used to improve ongoing programmes (learning programmes) and prepare future FPs. | |

|  |  |
| --- | --- |
| 17. | This report covers the relevant arrangements for FP4 (1995-1998), FP5 (1999-2002) and FP6 (2003-2006). It also discusses the legal requirements currently in force under FP7 (2007-2013), but does not provide an assessment of the Commission's FP7 ‘Impact assessment and ex-ante evaluation’[(16)](#ntr16-C_2008026EN.01000101-E0016). Its focus is on the monitoring and ex-post evaluation of programmes, and not the ex-ante evaluation of project proposals or ex-post evaluation of the projects themselves. |

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 18. | The audit was carried out with the participation of the EFTA Board of Auditors and with the assistance of external experts. Audit evidence was collected through:   |  |  | | --- | --- | | — | audit work at the Commission (documentation review and staff interviews), involving the identification of legal and statutory requirements, a review of objectives and performance criteria, the analysis of more than 80 monitoring and evaluation reports and an analysis of the organisation of evaluation activities within the Commission, |  |  |  | | --- | --- | | — | interviews with more than 90 experts identified as stakeholders in Member States and associated countries (Czech Republic, Germany, Estonia, Spain, France, Italy, Luxembourg, Netherlands, Austria, Finland, Sweden, United Kingdom and Norway), |  |  |  | | --- | --- | | — | surveys with institutional stakeholders at the European Parliament's ITRE[(17)](#ntr17-C_2008026EN.01000101-E0017) and the Council's CREST[(18)](#ntr18-C_2008026EN.01000101-E0018) committees, |  |  |  | | --- | --- | | — | a consultation with more than 65 experts in the field of research evaluation in the USA and Canada as an international comparison group, |  |  |  | | --- | --- | | — | a review of academic literature on current trends in research evaluation, and |  |  |  | | --- | --- | | — | the discussion of the audit findings, conclusions and recommendations with research evaluation experts from Europe and North America during a series of facilitated focus group meetings in February 2007. | |

AUDIT OBSERVATIONS

Intervention logic, objective setting and performance indicators

|  |  |
| --- | --- |
| 19. | Effective evaluation and monitoring requires clear objectives and relevant performance indicators at policy, programme and project level against which the results of a public intervention can be assessed. Good practice in other EU budgetary areas and further afield suggests that these should be defined within the framework of an intervention logic. |

|  |  |
| --- | --- |
| 20. | Intervention logic can be defined as a set of hypothetical cause and effect linkages that describe how an intervention is expected to attain its global objectives[(19)](#ntr19-C_2008026EN.01000101-E0019). The Commission's guidelines for evaluation state that intervention logic provides ‘the conceptual link from an intervention's inputs to the production of its outputs and, subsequently, to its impacts on society in terms of results and outcomes’[(20)](#ntr20-C_2008026EN.01000101-E0020). This report uses this definition throughout. |

|  |  |
| --- | --- |
| 21. | The Court assessed the extent to which intervention logic was established for the FPs and its constituent programmes, thereby providing a sound basis for evaluation and monitoring. |

Intervention logic for the FPs has not been explicit

|  |  |
| --- | --- |
| 22. | Defining intervention logic should be a key element of programme design from the outset. Article 163 of the Treaty, stating the need to strengthen the scientific and technological base and become more competitive, should provide the starting point for such an intervention logic for the FPs (see paragraph 2). This would need to be made more specific with regard to the different scientific areas covered by the FPs and the different instruments used. |

|  |  |
| --- | --- |
| 23. | However, an explicit logic model explaining how the FPs and their constituent programmes contribute towards scientific and technological progress and innovation is lacking at all levels. |

|  |  |
| --- | --- |
| 24. | The absence of intervention logic for the FPs was also highlighted by the 2004 5YA panel, which stated that ‘… panels like ours are asked to fill a gap between, on the one side, evidence mainly collected at project level and, on the other side, the higher level socio-economic goals of research policy. However, at the moment the link is difficult to make due to the way the FP is planned. It lacks an explicit logic connecting the highest objectives to the specific research and knowledge goals’[(21)](#ntr21-C_2008026EN.01000101-E0021). |

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 25. | A different approach is taken in some Member States (e.g. The Netherlands, Finland) and the USA (see Figure 3), where defining intervention logic is much more commonplace and in Canada where it is mandatory for all programmes (see Annex III). Indeed the use of intervention logic in programme design there results in more focused and better structured programmes. It is also being introduced by the Commission in other Community policy areas (e.g. external aid)[(22)](#ntr22-C_2008026EN.01000101-E0022).  Figure 3  Definition of a hierarchy of objectives based on logic models — example of US agricultural research   |  |  |  | | --- | --- | --- | | Policy | Programme | Project | | Policy objective:  to contribute to the improved livelihood of farming families |  |  | | Purpose:  increased agricultural production, productivity and incomes among farming households | Programme objective:  to contribute to increasing agricultural production, productivity and incomes among farming households |  | | Expected result:  the use of improved agricultural technologies increased among targeted farmers | Purpose:  increased use of improved agricultural technologies by farmers (e.g. rice) | Project objective:  to contribute to increased use of recommended improved technologies | |  | Expected result:  recommendations for targeted farmers provided/disseminated | Purpose:  recommendations provided for improved technologies suitable for targeted farmers | |  |  | Expected results (e.g.):   |  |  | | --- | --- | | 1. | technologies identified based on farmers priorities |  |  |  | | --- | --- | | 2. | technologies generated and adapted |  |  |  | | --- | --- | | 3. | technologies verified in farmers fields | | | Source: US National Agricultural Research Council. | | | |

|  |  |
| --- | --- |
| 26. | Not codifying the underlying rationale for public intervention and the assumed causality links (according to which the programme is then planned in detail, executed and assessed) undermines the effective evaluation and monitoring of the FPs. This is particularly due to how programme objectives and performance indicators are defined. |

Without an explicit logic model it is more difficult to set coherent programme objectives

|  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| 27. | Not making the underlying rationale of the public intervention explicit makes it more difficult to set and structure programme objectives coherently. This is demonstrated by the fact that:   |  |  | | --- | --- | | — | it is not clear how the FPs' programme objectives relate to policy objectives, such as the overall objective in Article 163 of the Treaty, those stated in the Lisbon strategy and the Barcelona declaration (see paragraph 2 and 7), |  |  |  | | --- | --- | | — | the relationship between individual programme objectives, both at the different layers of the legislation (such as FP decision, specific programme decision and rules of participation and dissemination) and between the basic acts and the Commission's work programmes, is not always apparent. | |

|  |  |
| --- | --- |
| 28. | The Commission has attempted to set out operational objectives in the work programmes based on the programme objectives defined in the FP legislation. However, in almost all cases, these initiatives have failed to clarify the programmes' underlying logic and the objectives stated in the work programmes were not specific, measurable, achievable, relevant and timed (SMART). In 2003, this became a legal requirement under the Financial Regulation[(23)](#ntr23-C_2008026EN.01000101-E0023). |

|  |  |
| --- | --- |
| 29. | The extent to which policy objectives can be traced through the specific programmes, work programmes and calls for proposals to the individual projects co-financed by the Community is unclear. Unless these links are clarified, the contribution of the FPs and their constituent programmes to the achievement of these policy objectives cannot be assessed. The audit found that little or no convincing evidence about the achievement of longer term objectives, both at project and programme level, has been provided by the Commission in its evaluations (see paragraph 82). |

The absence of an explicit intervention logic and clear objectives make it more difficult to develop performance indicators

|  |  |
| --- | --- |
| 30. | The absence of an explicit logic model and clearly spelled-out programme objectives has also impeded the development of performance indicators for monitoring the achievement of programme objectives in terms of outputs, outcomes and impacts, to the extent that, to date, such ‘effectiveness’ indicators have not been defined at all for the FPs (see paragraph 73). |

|  |  |
| --- | --- |
| 31. | In this situation, the review of ongoing projects funded by these programmes and the ex-post evaluation of completed ones is also rendered more difficult and contributes to the difficulties in aggregating evaluations from the project to the programme level (see Annex I and paragraph 84). |

ABB/ABM as currently implemented is an unsuitable tool for performance measurement of the FPs

|  |  |
| --- | --- |
| 32. | Some indicators that track the efficiency of programme implementation (e.g. time-to-contract, time-to-payment) are found in the Commission's annual monitoring reports and also in the context of the Commission's Activity-Based Management (ABM) system. |

|  |  |
| --- | --- |
| 33. | However, up to now, the ‘research DGs’, even when implementing the same specific programme (or sub-programme), have not applied a standardised or comparable Activity-Based Budgeting (ABB) nomenclature and related indicators. Therefore, these indicators cannot be used to compare performance between and within programmes.  Box 2ABB/ABM as currently implemented by the ‘research DGs’in the ABB/ABM system, the FPs are covered by two different policy areas within which the FP activities are presented according to a different DG-specific nomenclature[(24)](#ntr24-C_2008026EN.01000101-E0024). This even applies to those specific programmes and sub-programmes implemented by more than one DG. ABM objectives were defined according to different underlying principles[(25)](#ntr25-C_2008026EN.01000101-E0025) and, as indicated by the Commission's Internal audit service[(26)](#ntr26-C_2008026EN.01000101-E0026), were mostly formulated in general terms, describing aspirations or activities. Indicators were generally input-oriented (e.g. number of meetings, etc.). Some regarded process efficiency (e.g. time-to contract, time-to-payment), but were not comparable between programmes and from one DG to the other due to the differences in the ABB nomenclature, in DG-specific assumptions (such as the beginning and end of specific administrative processes) and in the underlying population (e.g. type of grant agreement, type of payment) that was taken into consideration for their computation. |

Recommendations

|  |  |
| --- | --- |
| 34. | In future legislative proposals to the Council and the European Parliament, the Commission should set out an explicit intervention logic for the FP as a whole and its constituent parts. |

|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 35. | For FP7, the Commission should:   |  |  | | --- | --- | | — | clearly describe the objectives of the FP and its constituent specific programmes (and sub-programmes) and how they should be achieved, |  |  |  | | --- | --- | | — | clarify the link between the scientific and socio-economic objectives for each specific programme (and sub-programme), and |  |  |  | | --- | --- | | — | define relevant operational objectives. | |

|  |  |
| --- | --- |
| 36. | For its FP7 monitoring system, the Commission should put in place a limited, but balanced, set of performance indicators to measure the efficiency and effectiveness of programme implementation for each specific programme (and sub-programme). |

|  |  |
| --- | --- |
| 37. | The ‘research DGs’ should harmonise their ABM/ABB nomenclature to make performance indicators comparable both over time and, where possible, between specific programmes (and sub-programmes). |

Evaluation strategy

|  |  |
| --- | --- |
| 38. | An evaluation strategy provides the conceptual framework within which evaluation activities are designed, planned, executed and used. Good practice in other EU budgetary areas and further afield suggests that such a strategy should consider the main legal, organisational and methodological issues surrounding programme evaluation. This includes what evaluations are to be carried out, by whom and when, how data are to be collected, what methodological approaches are to be used and how findings are to be communicated and followed up. |

|  |  |
| --- | --- |
| 39. | Having such a strategy in place, and making sure that it is implemented, is a pre-requisite for embedding evaluation in the Commission's organisational culture and decision-making process. The Court recognises, however, that given the diversity of the scientific areas covered by the FPs, such a strategy should provide an appropriate degree of flexibility to each of the DGs responsible and should not be overly prescriptive. |

|  |  |
| --- | --- |
| 40. | The Court checked the extent to which the Commission had in place such an evaluation strategy for the FPs. |

Absence of a comprehensive evaluation strategy for the FPs

|  |  |
| --- | --- |
| 41. | For the FPs, the Commission first made public its approach to evaluation in the 1980s[(27)](#ntr27-C_2008026EN.01000101-E0027). This was updated in 1996, when the Commission informed the European Parliament and the Council of what it then regarded as the relevant underlying principles for monitoring and evaluation and set out its intended approach following the adoption of FP4[(28)](#ntr28-C_2008026EN.01000101-E0028). This, however, did not constitute a comprensive evaluation strategy as set out above (see paragraphs 38 and 39). Furthermore, from 1996 to 2006, the Commission did not fundamentally re-examine its approach to evaluation for FP5 and FP6, despite the fact that FPs changed both in scope and orientation (see paragraph 7). In particular, the ‘research DGs’ have not agreed on a consistent approach to the evaluation of the FPs and its constituent programmes beyond the legislative requirements adopted in 1999 (for FP5), in 2002 (for FP6) and 2006 (for FP7)[(29)](#ntr29-C_2008026EN.01000101-E0029). |

|  |  |
| --- | --- |
| 42. | For the current programming period, the absence of such a strategic approach to evaluation in this field is a particular problem in view of the joint implementation of FP7 by several DGs, the ERC and executive agencies, and the mandatory staff rotation on sensitive posts imposed by the Financial Regulation. It also stands in contrast with current practice in the area of external aid (which is the second largest area of direct management by the Commission). In this area the competent DGs have, since 2002, agreed on common multiannual evaluation strategies linked to programming periods (see paragraph 52). |

|  |  |
| --- | --- |
| 43. | This absence of a clearly formulated evaluation strategy also contrasts strongly with the situation in the USA and Canada, where having a strategy is required by the regulatory framework (see Annex III). |

|  |  |
| --- | --- |
| 44. | As illustrated below, there are significant differences in the coverage and intensity of evaluation across the FPs. There were also other weaknesses, such as coordination problems, differences in methodological approaches, timing problems and weaknesses in the communication and follow up of evaluation findings, discussed in later sections of this report. This reflects the absence of a comprehensive evaluation strategy for the FPs. |

Differences in the coverage and intensity of evaluation across the FP

|  |  |
| --- | --- |
| 45. | FPs are composed of a number of specific programmes (and sub-programmes) that may differ significantly in terms of their purpose, underlying dynamics and budgetary appropriations (for FP7, see Annex IV). Taken together, the specific provisions of the various FP decisions and the Financial Regulation[(30)](#ntr30-C_2008026EN.01000101-E0030) (which requires the evaluation of programmes and activities which entail significant spending) imply that a legal requirement to evaluate both the FP as a whole and most of its constituent specific programmes has been in force since FP4. |

|  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| 46. | The Commission has not established a rule for when specific FP programmes or sub-programmes are to be evaluated or a common threshold above which this must be done. This has led to inconsistent interpretations as to what should be evaluated, and at what level of detail:   |  |  | | --- | --- | | — | DG Information Society considered the IST programme to be a major spending programme in its own right and organised a full 5YA in 2004 in parallel with that carried out for the FP as a whole, |  |  |  | | --- | --- | | — | DG Research did not systematically organise such an assessment at specific programme level despite the budgets of several programmes under its responsibility being of a comparable size. | |

|  |  |
| --- | --- |
| 47. | As a result, for the 2004 5YA, significant parts of FP4 and FP5 programmes were not evaluated at all (corresponding to 30 % of all budgetary appropriations for FP4 and 50 % for FP5). This means that the Commission did not comply with the evaluation requirements specified in Article 27 of the Financial Regulation. |

|  |  |
| --- | --- |
| 48. | These divergent interpretations with respect to the minimum legal requirements regarding evaluation, and the resulting under-evaluation of FPs in the past, illustrate the lack of coordination and the effect of the absence of a strategic approach by the Commission to the evaluation of the FPs. |

Recommendations

|  |  |
| --- | --- |
| 49. | The ‘research DGs’ should urgently develop and agree on a common evaluation strategy for the FPs. Without being overly prescriptive, this strategy should entail a consistent approach with regard to the evaluation of the FPs and its constituent programmes. Such a strategy should also provide guidance with regard to the minimum level at which detailed evaluation must take place so as to take account of the specificities of each scientific field. |

Organising evaluation for the FPs

|  |  |
| --- | --- |
| 50. | The evaluation of the FPs needs to be supported by appropriate organisational arrangements, which should follow from the Commission's strategic orientation for the evaluation of the FPs. The Court checked the extent to which the organisational structure ensured that the evaluation system was appropriately coordinated, adequately resourced and also perceived by stakeholders as being sufficiently independent. |

Commission

FPs are jointly implemented by several DGs whilst responsibility for evaluation is decentralised

|  |  |
| --- | --- |
| 51. | Within the Commission, responsibility for evaluating expenditure programmes rests with the individual DGs (see paragraph 8) and evaluation units have been set up within each DG. These evaluation units are in charge of planning, coordinating and executing evaluations. |

|  |  |
| --- | --- |
| 52. | The absence of a specific coordinating body or a single entity for the evaluation of the FPs as a whole, such as a joint evaluation office for the ‘research DGs’, stands in contrast to other policy areas. In the area of external aid, a joint evaluation unit has been established for three DGs (Development, External Relations and EuropeAid) to take charge of the evaluation of the Commission's development programmes in third countries (see paragraph 42). |

|  |  |
| --- | --- |
| 53. | DG Research, via the Commission's internal ‘RTD evaluation network’, ensures an exchange of knowledge between the evaluation units of the different DGs, but does not have the authority to set minimum evaluation requirements. This mechanism has, however, been insufficient to ensure an appropriate coordination of the Commission's evaluation of the FPs. |

|  |  |
| --- | --- |
| 54. | The Commission revised its general approach to evaluation in 2000 and 2002, and again in 2007 (see paragraph 14). Although no major improvements were observed in the way evaluations of the FPs and their constituent programmes were planned, designed and implemented, in general terms, the ‘research DGs’ have complied with the formal Commission-wide requirements. |

The Commission's central services: standard setting, but no enforcement role

|  |  |
| --- | --- |
| 55. | The Commission's central services (in particular DG Budget and the Secretariat-General) provide methodological advice, define common standards within the Commission as a whole and monitor how the DGs and services apply the standards (see paragraphs 14 and 70). They cannot impose how evaluations are to be done in practice. |

|  |  |
| --- | --- |
| 56. | Experience in the USA and Canada indicates that enforcement by standard-setting bodies is instrumental there in ensuring the effective functioning of the evaluation system. In both countries, federal departments establish and enforce the framework within which a programme's success is ultimately judged (see Annex III). Whilst the background is different, their systems are nevertheless based on the same underlying principle that a body separate from the one implementing the programme should assess the design of the evaluation system in some detail and scrutinise its effectiveness. The same can be observed in some Member States (e.g. France[(31)](#ntr31-C_2008026EN.01000101-E0031), Spain[(32)](#ntr32-C_2008026EN.01000101-E0032), Germany[(33)](#ntr33-C_2008026EN.01000101-E0033)), where external bodies supervise evaluations carried out by funding agencies or even carry out these evaluations on their own. |

Panels

Panels of external experts: required by FP legislation since FP4

|  |  |
| --- | --- |
| 57. | Since FP4, all major evaluation exercises (and annual monitoring) of the FPs have had to be carried out by (or with the assistance of) panels of external experts (see Annex II). Despite difficulties in terms of finding appropriate experts, such panels have the potential to provide transparency, offer a degree of independence, facilitate the involvement of stakeholders and bring to bear knowledge of the FPs themselves, evaluation and the underlying scientific issues. |

|  |  |
| --- | --- |
| 58. | This was confirmed by the Court's surveys, which indicated that respondents saw the Commission's panel-based evaluation system as providing a guarantee of independence (see Figure 4).  Image |

Weaknesses of the panel approach as implemented by Commission

|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 59. | Despite this positive perception, as panel members and stakeholders pointed out to the Court during the audit, the Commission's use of panels for the evaluation of FPs has been characterised by the following weaknesses:   |  |  | | --- | --- | | — | evaluation panels are usually set up less than a year before their report is due and only convene for a limited number of meetings. This makes it difficult for panel members to influence the definition of the data and analysis required for their work. In addition, because panels are ‘ad hoc’ and temporary, knowledge about programmes cannot be easily accumulated and relevant issues cannot to be followed up over time. This is a particular weakness when (as for FP7) a number of major evaluation exercises are planned throughout the programming period (see Annex II). It also stands in contrast with the Commission's practice for its Expert Advisory Groups (EAGs) for the FPs, which are permanent bodies[(34)](#ntr34-C_2008026EN.01000101-E0034). Also in Canada[(35)](#ntr35-C_2008026EN.01000101-E0035) evaluation panels are permanent in order to guarantee a consistent and coherent approach (see Annex III), |  |  |  | | --- | --- | | — | when defining the terms of reference for panels, and specifying the evaluation questions, the Commission does not consult with external stakeholders, such as industry and major research organisations participating in the FPs. The 2004 monitoring panel has also observed this problem noting that ‘… potential users should be involved in the process at an early stage’[(36)](#ntr36-C_2008026EN.01000101-E0036), and |  |  |  | | --- | --- | | — | panels operate in a way that is different from ‘peer review’. Their members do not have the time to carry out evaluations of their own; they typically provide advice based on data and evidence made available to them from other sources. As a result of the timing constraints, panels have had only limited opportunities to commission additional studies needed to address the questions specified in their terms of reference. In practice, with few exceptions, only information provided by the Commission was available to the 5YA panels. | |

Resources

No resources specifically earmarked for the FP evaluation

|  |  |
| --- | --- |
| 60. | Funding for evaluation is provided on an annual basis, according to needs specified by the DGs implementing the FPs. It is currently sourced from a combination of Commission-wide resources from DG Budget and resources from the administrative part of the budget at the level of the DGs concerned. In addition, specific measures are funded as indirect actions from the operational budget of the FPs. This means that there is no amount specifically set aside for evaluation. |

|  |  |
| --- | --- |
| 61. | In 2006, the Commission-wide average for staff working in the area of evaluation and activity-based management was estimated at approximately 1 % of total staff[(37)](#ntr37-C_2008026EN.01000101-E0037). The corresponding figures for monitoring and evaluation activities of the RTD framework programmes at DG Research and DG Information Society were 1,2 % and 1,3 % respectively. In the case of DG Research, more than two thirds of these staff resources were, however, allocated to the operational programme departments rather than to the evaluation unit, and evaluation was only one of many tasks assigned to them. |

Funding for FP evaluation lower than recommended by the Commission

|  |  |
| --- | --- |
| 62. | Resources devoted to evaluation can also be expressed as a percentage of the overall budgetary appropriations allocated to the programme. Figures obtained during the audit indicate that the practice for research programmes within the EU Member States lies between 0,1 % and 2 %. The Commission's guidelines suggest that 0,5 % of a programme's budget should be devoted to evaluation[(38)](#ntr38-C_2008026EN.01000101-E0038). However, for larger programmes, economies of scale mean that a smaller share of their budgets could suffice. |

|  |  |
| --- | --- |
| 63. | A comparison between the Commission's figures for expenditure on evaluation in the 2003-2006 period and the budgetary appropriations for each Directorate-General shows that the overall average runs at around 0,15 %, which is significantly below the level suggested by the Commission. A closer analysis for the year 2006 reveals that DG Information Society spent approximately four times more than DG Research on evaluation compared to the appropriations under their respective responsibilities. In addition, at DG Research, only 0,7 % of one full time official is devoted to evaluation per million euro annual budget. By comparison, DG Information Society devotes double that amount. |

|  |  |
| --- | --- |
| 64. | As illustrated in this report, this additional effort devoted by DG Information Society has resulted in a more comprehensive evaluation coverage (see paragraphs 13 and 46) and, as shown in the sample of evaluation studies reviewed by the Court, the use of methodologies that are better suited to addressing evaluation needs (see paragraph 84). |

|  |  |
| --- | --- |
| 65. | A cost estimate made by the Court based on data provided by the Commission shows that an evaluation system based on the proposals made in this report would, on average, represent less than 0,4 % of the FP7 budgetary appropriations. This would cover the costs of panels, sub-contracted studies, staff of both the joint evaluation office and the current DG-specific evaluation units and other miscellaneous costs[(39)](#ntr39-C_2008026EN.01000101-E0039). |

Recommendations

|  |  |
| --- | --- |
| 66. | The ‘research DGs’ should consider setting up a joint evaluation office in addition to the existing evaluation units within each DG. In line with current practice in other policy areas, this office could coordinate the Commission's evaluation activities for the FP as a whole. Such common support functions have been recommended in the past for the DGs implementing the FPs[(40)](#ntr40-C_2008026EN.01000101-E0040) and may become even more important in the future with the creation of the ERC and the delegation of programme implementation to executive agencies. |

|  |  |
| --- | --- |
| 67. | Evaluation panels should continue to be used by the Commission to assist it in its evaluation activities. However, the Commission should consider how to align the panel structure with the overall FP programme structure. This could be done by having panels for specific programmes and, where appropriate, sub-programmes (see Annex IV). In addition, these panels should be set up sufficiently early in order to provide effective support to the Commission, and continue to do so throughout the programming period and beyond. |

|  |  |
| --- | --- |
| 68. | The Commission should reconsider the resources to be used for evaluation of the FPs and their constituent programmes. |

Evaluation methodologies and techniques

|  |  |
| --- | --- |
| 69. | Effective evaluation requires the use of appropriate methodologies and techniques. The Court checked the extent to which specific guidelines for the evaluation of the FPs existed, whether information needs were properly analysed on a timely basis and whether relevant data were made available to evaluators. Considering the evaluation studies carried out by (or on behalf of) the Commission, the Court reviewed whether the techniques used for the individual studies provided a sufficient basis for addressing the evaluation questions set for the 5YAs. The potential contribution that might be made to the evaluation of the FPs by national-level evaluations was also considered. |

Deploying FP-specific standards and guidelines

|  |  |
| --- | --- |
| 70. | DG Budget has issued several Communications setting out its evaluation policy and providing rules for its implementation and has established a set of Commission-wide standards (see paragraph 14 and 55)[(41)](#ntr41-C_2008026EN.01000101-E0041). For other policy areas, the competent DGs have also developed specific evaluation guides[(42)](#ntr42-C_2008026EN.01000101-E0042). |

|  |  |
| --- | --- |
| 71. | This is, however, not the case for the FPs. There are a number of methodological guides and toolboxes for evaluation activities in the field of research that have been published (or funded) by the Commission[(43)](#ntr43-C_2008026EN.01000101-E0043). However, these neither provide specific guidance on how to carry out evaluations for the FPs nor define how key concepts or terminology are to be used by Commission staff or external experts. This has contributed towards the problems discussed below. |

Data needs and data collection

Insufficient analysis of information needs

|  |  |
| --- | --- |
| 72. | Insufficient attention has been paid by the ‘research DGs’ to understanding what data is required for evaluation and monitoring purposes. Rather than carrying out a systematic and comprehensive analysis of information needs, the Commission has specified data requirements late in the programming period and without due regard to overall evaluation needs. The FP6 socio-economic questionnaire was the only common attempt among DGs to agree on such data (see paragraph 76). However, this questionnaire was finalised in 2004 (three years after the start of the programming period) and primarily covers aspects such as gender participation and environmental impacts, rather than scientific outputs (such as intellectual property issues, publications, etc.), outcomes or impacts. |

|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 73. | The assessment of programmes relies heavily on the availability of three kinds of data:   |  |  | | --- | --- | | — | data collected according to effectiveness indicators, the analysis of which provides intelligence about progress towards programme objectives. The lack of an intervention logic referred to above means that it has been difficult to define such indicators (see paragraph 30). In practice, the data that were collected were of little relevance for measuring the achievement of programme objectives, |  |  |  | | --- | --- | | — | data about programme management (according to efficiency indicators), and |  |  |  | | --- | --- | | — | data about participants (such as the participation rates of major research organisations). | |

|  |  |
| --- | --- |
| 74. | In particular, the latter two should be available at the Commission. However, due to problems with respect to the Commission's internal IT systems, which are the source of most of the data on programme management and participants[(44)](#ntr44-C_2008026EN.01000101-E0044), significant problems have been experienced with this in practice[(45)](#ntr45-C_2008026EN.01000101-E0045). |

|  |  |
| --- | --- |
| 75. | This is illustrated by the considerable delays in making Article 173 Reports and Annual Monitoring Reports available (up to 16 months late for Article 173 Reports[(46)](#ntr46-C_2008026EN.01000101-E0046) and up to seven months late for Annual Monitoring Reports in the period between 2002 and 2005). As this information was consistently late, it was of limited use. |

Ineffective approach to sourcing evaluation data from participants

|  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| 76. | Since FP5, the Commission has systematically required participants, by contract, to provide data needed to measure the achievement of programme objectives in terms of outputs, outcomes and impacts, as follows:   |  |  | | --- | --- | | — | under FP5, participants were required to submit a ‘Technological Implementation Plan’ together with a final report after the project had been completed. These documents were often of poor quality or were not submitted at all, |  |  |  | | --- | --- | | — | under FP6, they are required to produce a ‘Plan for Disseminating and Using Knowledge’ and must respond to a standardised questionnaire concerning socio-economic issues when submitting their annual technical reports. | |

|  |  |
| --- | --- |
| 77. | These contractual requirements to provide data have placed unnecessary administrative burdens on participants. For example, with respect to the FP6 annual questionnaire, data are collected much more frequently than needed and from the entire population of projects when a representative sample would suffice. This is likely to lead to ‘evaluation fatigue’. Moreover, this information has been collected at the wrong time; many of the underlying issues for which data are collected are medium or longer-term in nature so that data collection within the contractual duration of the project is by definition premature[(47)](#ntr47-C_2008026EN.01000101-E0047). This is why only limited practical use has been made of the information provided by these tools for evaluations under FP5 and FP6. |

|  |  |
| --- | --- |
| 78. | However, none of the systems in place provides for information to be collected at the end of (or shortly after) a project's lifespan in a standardised manner. Stakeholders interviewed suggested that such information might include the achievement of project objectives, intended (and unplanned) results and the perception of Community added value by participants. Such information could also form the basis for external reviews of completed projects[(48)](#ntr48-C_2008026EN.01000101-E0048). |

|  |  |
| --- | --- |
| 79. | Together with a more extensive use of scientometric approaches (such as citation analysis) to assess project results (see paragraph 83), this could provide a basis for the Commission to move to a more results-based financing system for RTD activities co-funded by the Community[(49)](#ntr49-C_2008026EN.01000101-E0049). |

Other data sources inadequately used

|  |  |
| --- | --- |
| 80. | Insufficient use is made of other existing sources of data within the Commission and the Member States (e.g. databases of national RTD programmes and funding agencies and the ‘Community Research & Development Information Service’ (CORDIS) operated by the Commission's Publication Office[(50)](#ntr50-C_2008026EN.01000101-E0050)). |

Evaluation studies carried out by the Commission

Reports reviewed by the Court

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 81. | The Court reviewed 86 evaluation studies carried out by (or on behalf of) the Commission since 2000. These included all 36 studies forming the basis for the 5YA published in 2004 and the further 50 studies completed since that date. Of these, only 23 studies can be considered to be genuine evaluations of the FP and its programmes (see Figure 5).  Figure 5  Overview of evaluation studies carried out by (or on behalf of the Commission) in the period 2000 to 2006   |  |  |  | | --- | --- | --- | | Type of studies | Number | Description | | Evaluations of FP activities | 23 | Evaluation studies   |  |  | | --- | --- | | — | based on recognised methodologies[(51)](#ntr51-C_2008026EN.01000101-E0051), |  |  |  | | --- | --- | | — | make specific reference to programme objectives and to the extent to which programme objectives have been met. | | | National impact assessments undertaken at Member State level | 12 | Studies carried out by Member States which address mainly national participation rates, rather than concluding about programme objectives. | | Methodological studies | 5 | Studies organised by the Commission to assist in the development of evaluation methodologies. | | Studies of the European RTD landscape | 2 | Studies addressing contextual analyses on the research scene in Europe, but not the FPs. In particular no explicit reference is made to the effect that the FPs have on national research and innovation systems. | | Non-analytical descriptive and statistical documents | 33 | Documents which typically address FP activities in a very indirect fashion, often describing activities rather than analysing them. | | Other documents | 11 | A range of documents that consider issues of general policy interest, without examining the FPs in that context. | |

Evaluations focusing on monitoring issues, rather than outcomes and impacts

|  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| 82. | Two main observations result from the Court's meta-analysis of these specific evaluation studies:   |  |  | | --- | --- | | — | whilst many of the studies attempt to conclude against programme objectives, these conclusions are presented in a vague manner, leaving the reader unconvinced that the evaluation has actually addressed this issue. This is often because the lack of any clear programme objectives critically undermines the process (see paragraph 28), |  |  |  | | --- | --- | | — | rather than evaluating the outcomes and impacts of the FPs or their specific programmes (and sub-programmes), many of the documents are descriptive, and, at best, concentrate on immediate project outputs and project management matters. These reports are unable to provide convincing evidence with regard to outcomes or impacts. | |

Insufficient focus on the development of innovative methodologies

|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 83. | The use of traditional methodologies and techniques (in particular surveys, document review and interviews) dominates for evaluations carried out in the period 2000 to 2006 (see Figure 6). Furthermore, a number of other sophisticated, but nevertheless well-established, evaluation methodologies and techniques were not (or only very rarely) employed, in particular:   |  |  | | --- | --- | | — | scientometric approaches (such as citation analysis)[(52)](#ntr52-C_2008026EN.01000101-E0052), |  |  |  | | --- | --- | | — | quantitative models (such as cost-efficiency, cost-effectiveness and cost-benefit)[(53)](#ntr53-C_2008026EN.01000101-E0053), or |  |  |  | | --- | --- | | — | actor (or behavioural) models (which focus on the perspectives of those involved in an intervention, such as the characteristics of research networks and changes thereto over time). |   Image |

|  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 84. | Evaluation studies making an attempt to address some of the methodological difficulties of assessing public intervention in research (see Annex I) were used only in a limited number of cases:   |  |  | | --- | --- | | — | recently two studies were completed on the subject of intervention logic[(54)](#ntr54-C_2008026EN.01000101-E0054), |  |  |  | | --- | --- | | — | only three studies have been identified which attempt to assess the effect that Community funding has had on the pan-European networking behaviour of research organisations[(55)](#ntr55-C_2008026EN.01000101-E0055). This is despite the fact that the assumption of positive networking effects is at the heart of the ERA concept, |  |  |  | | --- | --- | | — | in only one case has an attempt been made to draw conclusions about the impact achieved at programme (or sub-programme) level by combining the aggregation of project level evaluation analyses with other evaluation techniques[(56)](#ntr56-C_2008026EN.01000101-E0056), and |  |  |  | | --- | --- | | — | no evidence has been found for the use of other methodologies, such as ‘counterfactual’ (or ‘control group’) approaches. This is an approach that would be particularly useful for illustrating the added value of programmes at Community level compared to not intervening at all (or intervening at national or regional level). | |

|  |  |
| --- | --- |
| 85. | Stakeholders interviewed by the Court were of the opinion that the usefulness and relevance of the Commission's evaluations could be improved. This demonstrates the need for further methodological development so that the evaluation system can evolve to take account of the changing character of the FPs (see paragraph 7)[(57)](#ntr57-C_2008026EN.01000101-E0057). |

Challenging issues raised in 5YAs could not be addressed on the basis of the Commission's specific evaluation studies

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 86. | This is also illustrated by the significant problems faced by the 2004 5YA panel[(58)](#ntr58-C_2008026EN.01000101-E0058) in answering the evaluation questions suggested in the terms of reference. Many of these challenging questions could not be examined at all, due to the absence of appropriate studies carried out by (or on behalf of) the Commission beforehand and the impossibility of the panel members to commission such studies in good time (see Figure 7).  Figure 7  Addressing evaluation questions in the 2004 5YA   |  |  |  | | --- | --- | --- | |  | Evaluation question | Addressed in report? | | IMPLEMENTATION | Were the activities carried out efficiently? | partially | | Were the activities cost effective? | partially | | Did the activities constitute the best way of achieving the objectives set? | no | | Were the overall legal framework (including rules for participation and contracts), policy instruments and the modalities for implementation clear, appropriate and effective? | no | | Were the level of funding and other available resources adequate? | no | | Were the targeted industrial and research communities, including SMEs, able to respond appropriately? | yes | | ACHIEVEMENTS | Did the activities attain their respective objectives and to what extent were there unexpected results? | no | | What are the major results in particular in terms of scientific, technological, socio-economic and environmental outputs, in terms of international co-operation, knowledge transfer and innovation, pre-normative activities, accessibility, dissemination and uptake of research, human resources development, mobility and training, and in terms of supporting and enhancing co-ordination of research activities? | no | | Were the results and their effects and impacts globally satisfactory from the point of view of direct or indirect beneficiaries and stakeholders? | no | | Were the relevant industrial and research communities, including SMEs, addressed satisfactorily? | yes | | How and how far have the activities contributed to improved EU research competitiveness at international level? Did EU research attain leadership in specific areas? | partially | | How and how far have activities contributed to EU policies in general and to the EU's strategy for sustainable development? | no | | Is there evidence of a structural change, including in particular networking, integration and coordination of research, at a national or at an international level as a result of Community research activities? | no | | Did the programmes provide value for money? Did the activities have lasting impacts? | no | | EVOLUTION AND FUTURE PERSPECTIVES | How did Framework programmes evolve in terms of rationale, objectives, thematic priorities, balance between bottom-up and top-down priorities and between fundamental and applied research, instruments, European added value and other impacts, in particular in the light of the emergence of the European Research Area (ERA) concept, the Lisbon objectives and major international economic and research benchmarks? | yes | | How did the Framework Programme achieve European added value? | yes | | Can European added value be achieved through other international or national mechanisms? | no | | Are the programmes' policy objectives, priorities, instruments and lifecycle appropriate for the future? | yes | |

|  |  |
| --- | --- |
| 87. | In addition, it is noted that only for those aspects where clear and quantifiable objectives were defined in the programmes (such as the share of small and medium-sized enterprises participating), were the panel members able to answer the questions specified in the terms of reference. |

Evaluations carried out at national level for the FPs

|  |  |
| --- | --- |
| 88. | Several Member States and associated countries carry out their own national evaluations of the FPs, which address relevant and interesting questions from their own perspective[(59)](#ntr59-C_2008026EN.01000101-E0059). In the last eight years, 14 such studies have been produced for nine Member States and one associated country. |

|  |  |
| --- | --- |
| 89. | Whilst the Commission encouraged standardised approaches where possible, these national impact studies are very heterogeneous[(60)](#ntr60-C_2008026EN.01000101-E0060). As a result, since they do not follow a harmonised methodology, possibilities to aggregate, combine or compare results are limited. Moreover, given their national character, these studies can only partly capture the European cooperative dimension. |

|  |  |
| --- | --- |
| 90. | In the 2007 ERA ‘Green paper’ the Commission argued that it was important that accurate analyses should be available at ERA level on the impact and effectiveness of research activities and policies, including those of the Member States and the EU. It also recognised that current arrangements were not adequate to address this challenge because each level of intervention performed evaluations separately from the others[(61)](#ntr61-C_2008026EN.01000101-E0061). |

Recommendations

|  |  |
| --- | --- |
| 91. | The Commission should draw up a manual for the evaluation of activities funded by the FPs, providing definitions of key concepts and terminology and identifying a range of appropriate methodologies for certain evaluation activities. This manual should be based on existing guides and toolboxes and should be updated on a regular basis. |

|  |  |
| --- | --- |
| 92. | The Commission should define its information needs, taking into account the need to limit the administrative burden on participants. |

|  |  |
| --- | --- |
| 93. | The Commission should provide monitoring information on the implementation of the FPs in a timely manner. As required by the FP7 ‘Rules for Participation’, problems relating to the Commission's IT systems for managing data on programme management and participants should be addressed urgently. |

|  |  |
| --- | --- |
| 94. | Participants' reporting obligations towards the Commission should be reduced significantly and surveys carried out on a sample basis only. The contractual requirement to submit indicator-based impact data for each project should be replaced by a short self-assessment upon project completion. |

|  |  |
| --- | --- |
| 95. | The Commission should make more extensive use for its evaluations of existing complementary sources of relevant data (both within the Commission and in Member States). |

|  |  |
| --- | --- |
| 96. | The Commission should employ the full range of evaluation techniques available. In addition, it should support the development of innovative approaches to address the methodological difficulties inherent to evaluating public intervention in research (see Annex I) and use these approaches in its own evaluations. |

|  |  |
| --- | --- |
| 97. | The Commission should consult with Member States on how to maximise the benefits of national-level evaluations of the FPs. |

Disseminating and using evaluation findings

|  |  |
| --- | --- |
| 98. | Evaluation should provide relevant information and analysis that can be effectively used for programme management and policy making. In particular, evaluation should provide the basis for the informed reflection on the strengths and weaknesses of a public intervention that is necessary for learning to take place. The Court checked the extent to which evaluations were properly timed, findings were communicated and disseminated to stakeholders outside the Commission and recommendations were taken into account by the Commission to revise ongoing programmes (learning programmes). |

Timing of evaluations did not take into account the type of information that could realistically have been expected

|  |  |
| --- | --- |
| 99. | The Court recognises that certain types of analysis require a long-term perspective (e.g. the evaluation of outcomes and socio-economic impacts) and that some aspects are related to a specific programming period (e.g. programme objectives, even within a given scientific field), whereas others are not (see Figure 8). However, the Commission has not adopted an approach in its evaluation activities that takes these time horizons into consideration. |

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 100. | The legislation has required the Commission's main evaluation activities to focus exclusively on the five years preceding the 5YA thereby covering parts of two successive programming periods. As a result, medium and long-term outcomes and impacts have barely been addressed. As a consequence, the 5YAs have generally focused mostly on monitoring aspects, short-term programme design issues and general observations about the orientation of the FPs. The Court welcomes the fact that FP7 legislation no longer requires 5YAs, or any other pre-defined period of coverage.  Figure 8  Monitoring and evaluation issues according to different time horizons   |  |  |  |  | | --- | --- | --- | --- | | Monitoring of programme implementation  (on-going) | |  |  | | --- | --- | | — | Programme management issues (e.g. time-to-contract, time-to-payment, subscription and success rates, budgetary execution, funding rates, etc.) | | | Evaluation  Short term  (say, at the earliest after 1-2 years) | |  |  | | --- | --- | | — | Programme management issues (e.g. efficiency of administrative procedures) |  |  |  | | --- | --- | | — | Programme design issues (e.g. accessibility and flexibility of instruments, barriers to participation, implications of non-success in calls for proposals, etc.) |  |  |  | | --- | --- | | — | Analysis of participation profile (mainly quantitative at this stage) | | | Evaluation  Medium term  (say, after 7 years) | |  |  | | --- | --- | | — | Programme design issues (e.g. effectiveness of instruments) |  |  |  | | --- | --- | | — | Analysis of participation (combining qualitative and quantitative aspects, e.g. research networks analysis, impacts on the behaviour of researchers, etc.) |  |  |  | | --- | --- | | — | Analysis of project outputs [(62)](#ntr62-C_2008026EN.01000101-E0062) (e.g. publications, conference papers, patents, licences, prototypes, standards, trained scientists, etc.) | | | Evaluation  Long term  (say, after > 10 years) | |  |  | | --- | --- | | — | Assessment of outcomes and impacts | | | Source: European Court of Auditors. | | |

|  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| 101. | The shortcomings of the 5YAs in assessing the outcomes and impacts of the FPs has also been noted by the Commission's experts:   |  |  | | --- | --- | | — | the 2004 5YA panel noted that ‘… evaluation should cover and give sufficient attention to both long-term and short-term issues’ and ‘… resources could be released […] to allow more ambitious long-term evaluation studies that could explore the structural and wider socio-economic issues in a more systematic way’[(63)](#ntr63-C_2008026EN.01000101-E0063), |  |  |  | | --- | --- | | — | a study commissioned by DG Research found that ‘… current EU RTD evaluation practices (comprising continuous monitoring, five year assessments and mid term evaluation) are characterised by strong focus on monitoring compared to impact assessment’ and ‘… in order to shift demand and expectation […] towards socio-economic impact there would need to be a new evaluation and monitoring framework that obliged a greater effort at impact level rather than monitoring of outputs’[(64)](#ntr64-C_2008026EN.01000101-E0064). | |

|  |  |
| --- | --- |
| 102. | The issue of timeliness also affects specific evaluations required by the legislation. For example, the 2004 Marimón report, which analysed the new instruments introduced in FP6 (2003-2006), was required too early, and as a result, there was little or no practical experience and empirical evidence on which to base its conclusions[(65)](#ntr65-C_2008026EN.01000101-E0065). |

Communication and dissemination

|  |  |
| --- | --- |
| 103. | On its own, the Commission can develop and adapt operational work programmes according to short-term cycles, subject to approval by the relevant programme committee. This could represent an opportunity for adjustments to be made to programme management, and also for reallocations of budgets within sub-programmes. No evidence was found that the Commission makes use of this possibility (see paragraph 107). |

|  |  |
| --- | --- |
| 104. | Beyond this, given the way the FPs are designed and adopted, it is all but impossible for the Commission to make significant changes, such as modifications of programme objectives or the reallocation of budgets between sub-programmes. Only the legislator can enact such wide-ranging modifications during a revision of the FP legislation (such as in 2010 for FP7) or the adoption of the subsequent FP. For this reason, the legislation has typically stipulated that the main evaluation exercises be timed so that their results could be most usefully taken into account for policy making. |

|  |  |
| --- | --- |
| 105. | Therefore, some of the main users of the Commission's evaluations are the ‘research DGs’ themselves[(66)](#ntr66-C_2008026EN.01000101-E0066). However, the FP legislation also explicitly requires evaluations to be communicated to the European Parliament and the Council. |

|  |  |
| --- | --- |
| 106. | In the Court's survey of the European Parliament's ITRE committee and the Council's CREST committee, a majority of respondents, however, indicated that they did not feel sufficiently informed about the implementation and the results of the FPs (see Figure 9)[(67)](#ntr67-C_2008026EN.01000101-E0067).  Image |

Follow-up of evaluation findings

|  |  |
| --- | --- |
| 107. | In its Annual Evaluation Reviews, the Commission provides a summary of findings and actions taken stemming from its evaluations[(68)](#ntr68-C_2008026EN.01000101-E0068). With regard to the follow-up, the Commission has also provided formal replies to the findings and recommendations of the 5YAs and the specific evaluations required by the legislation and, to the extent possible, has taken appropriate measures to address the weaknesses identified therein. This was, however, not the case for many of the other evaluation reports reviewed and no evidence was found that their findings and recommendations were taken into account for amendments to work programmes. Similarly, the DGs' ABB budgetary statements and their Annual Activity Reports do not indicate the extent to which evaluation findings were acted upon. |

Recommendations

|  |  |
| --- | --- |
| 108. | The Commission should establish the type and scope of evaluation that can reasonably be expected for the dates specified in the legislation (2008, 2010 and 2015), explain how evaluations can be used to adapt programmes (learning programmes) and specify what contribution evaluations can make to policy decisions. In particular, the Commission should clarify its intended approach with regard to the evaluation of the longer-term results of past and present FPs, given that such results may take seven to 10 years to come to fruition (see Figure 8). |

|  |  |
| --- | --- |
| 109. | The Commission should review its practice on communicating and disseminating its evaluation findings to ensure that stakeholders receive the necessary information. In particular, the ‘research DGs’ should provide a formal response to all evaluations and make these studies public in a timely manner, together with their replies. The ‘research DGs’ should also report in their ABB budgetary statements and their AARs on the follow-up given and make use of recommendations when updating work programmes and designing future programmes. |

OVERALL CONCLUSION

|  |  |
| --- | --- |
| 110. | The Court recognises that evaluating RTD programmes, and in particular assessing their long-term results, is inherently difficult and best practice is hard to define. |

|  |  |
| --- | --- |
| 111. | In the Court's view, evaluation is essential for programme management, for future planning and for policy making. Evaluation also provides a basis for accountability. It provides valuable information about what intended (or unintended) results have been achieved, the extent to which programme objectives have been reached, the relevance and utility of programmes and the efficiency and effectiveness of programme implementation. As it is the European Parliament and the Council that decide upon general orientation, budgetary matters and programme design for the FPs, a transparent, credible and robust evaluation system is needed to provide them with the appropriate information. |

|  |  |
| --- | --- |
| 112. | The Commission established an overall evaluation framework in 2000 and 2002 and subsequently revised it in 2007. Although during the period audited by the Court no major improvements were observed in the way evaluations of the FPs and their constituent programmes were planned, designed and implemented, in general terms, the ‘research DGs’ have complied with the formal Commission-wide requirements. |

|  |  |
| --- | --- |
| 113. | For the period covered by this report, as required in the FP legislation, the ‘research DGs’ had an evaluation system in place and the Commission can point to a sizeable body of evaluation studies. However, the fact remains that little or nothing is known about the achievement of programme objectives and the results of the FPs. This is because evaluations have generally focussed on short-term issues of programme implementation. As a result, the Commission's evaluations were of limited usefulness to policy-makers, stakeholders and even to the Commission itself. |

|  |  |
| --- | --- |
| 114. | Given the importance of evaluation for programme management and policy-making, the Commission's approach to evaluating the FPs and their constituent programmes needs to be re-examined in view of new political challenges, increased funding, a broadening of the orientation of Community research policy and the recent Commission reform. This was also recommended by EURAB in its 2007 report. |

|  |  |
| --- | --- |
| 115. | In the Court's view, the use of an explicit intervention logic in future FP programme design would lead to more focussed and better structured programmes. It would also contribute to the Commission's ‘better regulation’ policy, which aims for a better designed, simpler, more effective and better-understood regulatory environment[(69)](#ntr69-C_2008026EN.01000101-E0069). |

|  |  |
| --- | --- |
| 116. | Evaluation of the FPs and their constituent programmes would be more effective if programme objectives and indicators were clearer from the outset, a robust, but not overly prescriptive, evaluation strategy were agreed upon by the ‘research DGs’ implementing the FPs and more advanced evaluation methodologies, including quantitative approaches, were used. |

|  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| 117. | The Court also noted a number of respects in which there is scope for the Commission to reconsider its existing organisational arrangements for the evaluation of the FPs. There may be advantages in:   |  |  | | --- | --- | | — | involving evaluation panels earlier in the evaluation process, thereby providing assistance to the Commission throughout the programming period and beyond, and increasing specialisation by having panels for specific programmes and, where appropriate, sub-programmes, |  |  |  | | --- | --- | | — | supplementing the existing DG-specific evaluation units with a joint evaluation office in charge of coordinating the FP evaluations. | |

|  |  |
| --- | --- |
| 118. | The recommendations contained in this report are based on either existing practices within the Commission or international good practice. With regard to resources, it is clear that the introduction of the suggested approach to programme evaluation would entail higher costs. However, in the Court's view, it is most likely that this will produce a positive pay-off of at least the same order. |

This report was adopted by the Court of Auditors in Luxembourg at its meeting of 22 November 2007.

For the Court of Auditors

Hubert WEBER

President

---

[Top](#document1)