Source: EURLEX
Language: en
Format: md

_**\ s**_

COMMISSION OF THE EUROPEAN COMMUNITIES

Brussels, 22.05.1996
COM(96)220 final

COMMUNICATION FROM THE COMMISSION

TO THE COUNCIL AND THE EUROPEAN PARLIAMENT

INDEPENDENT EXTERNAL MONITORING AND EVALUATION OF

COMMUNITY ACTIVITIES IN THE AREA OF RESEARCH AND

TECHNOLOGICAL DEVELOPMENT

SUMMARY

The Commission is continually seeking to demonstrate and enhance the credibility and
efficiency of its efforts in relation to evaluation of Community RTD actions. In the light of
the recent legislative Decisions on the fourth Framework Programme and the related specific
programmes, which prescribe a complex evaluation scheme and an active role for independent
external experts in this scheme, the Commission decided to take a fresh look at Community
RTD evaluation efforts. The Scientific and Technical Research Committee (CREST) was
consulted for advice; subsequently, CREST established an Evaluation Sub-Committee. The
Council's Research Group and Atomic Questions Group as well as the European Parliament's
Committee on Research, Technological Development and Energy (CERT) have been informed
of this development.

This Communication summarizes the evaluation required by legislative Decisions and outlines
a rationalized approach, taking into account the CREST advice. The approach will produce
results in a timely, efficient and cost-effective manner. It concentrates on the continuous
monitoring of specific programmes and the Framework Programme and their five-year
assessment. Regular reporting will be made once a year, correlated with the publication of
the Annual Report requested under Article 130p of the Treaty on European Union. The
approach has been introduced on an experimental basis in 1995 and will be reviewed after
three years of implementation.

The approach builds upon the solid foundation of experience already acquired by the
Commission from past evaluations and takes into account a recent initiative on concrete steps
towards best evaluation practice in the Commission. The main feature of the approach is a
further development towards coherent monitoring and evaluation of Community RTD actions.
In particular, the approach:

(i) introduces continuous monitoring and develops through programme evaluation to S&T
policy evaluation;
(ii) spells out the implementation principles;
(iii) involves appropriate assistance from independent external experts; and
(iv) should produce results in time for discussion on future programme Decisions.

The Commission is responsible for the monitoring and evaluation of Community RTD efforts.
However, the involvement of outside experts and bodies and the openness of procedures will
enhance the _independence_ and _transparency_ of monitoring and evaluation. In addition, a new
level of _accountability_ is introduced through publication of the Commission's responses to
recommendations produced by the five-year assessment panel of the Framework Programme.
This will all enhance the _credibility_ of Community RTD evaluation.

Through this Communication, the Commission invites the Council and the European
Parliament to take note of this rationalized approach which takes account of the current
legislative requirements for monitoring and evaluation.

1. **INTRODUCTION**

The Commission's White Paper "Growth, Competitiveness, Employment", COM(93)700
final, identifies increased RTD effort and its adaptation to new market conditions as one
of the key means to improve the industrial and technological performance of Europe.
Evaluation - both the evaluation of the programmes (their management, monitoring
quality, efficiency of internal procedures, etc.) and the evaluation of results (increase in
scientific and technical knowledge, dissemination and optimization of results, industrial
exploitation of RTD results and their relation to innovation processes, measurement of
long-term economic and social effects, etc.) - plays an important role in this regard by
contributing to: the efficient implementation of RTD efforts; coherence with all relevant
policies; and best use of public resources.

The legislative Decisions on the fourth Framework Programme [1] and its specific
programmes, require the Commission to implement a complex evaluation scheme (see
Point 3 below). In this Communication the Commission presents a rationalized approach
which will produce results in a timely, efficient and cost-effective manner.

2. **EVOLUTION OF COMMUNITY RTD EVALUATION EFFORTS**

The systematic evaluation of Community RTD activities was confirmed with the Plan of
Action of 1983 [2] and has since evolved further [3] . The use of external experts has been a
key feature in promoting the independence of evaluation.

In 1993, the Council and the European Parliament invited the Commission to continue
to improve the credibility of the evaluation of Community RTD programmes while
ensuring its independence. Subsequently, the legislative Decisions on the fourth
Framework Programme included additional requirements concerning evaluation.

In 1994, the Commission Services involved in the implementation of the Framework
Programme, prepared a reflection document as a first step in response to the Council and
the European Parliament in this regard. The document was transmitted to CREST, CERT
and the European Science and Technology Assembly (ESTA).

In 1995, CREST produced advice [4] which included the establishment of a CREST
Evaluation Sub-Committee. The Research Group and the Atomic Questions Group
(Council), as well as CERT (Parliament), have been kept informed of this development.

On 21 March 1995 the Commission approved a programme [5] for "Sound and Efficient
Financial Management (SEM 2000)", divided into three phases: consolidation within the

In this document, the term "fourth Framework Programme" comprises Community activities resulting from two Decisions: J^ Decision N°
1110/94/EEC concerning the 4th Framework Programme of the European Community activities in the field of research and technological
development and demonstration (1994-1998); and, 2L Decision N° 94/268/Euratom concerning a framework programme of Community
activities in the field of research and training for the European Atomic Energy Community (1994-1998).

Council Resolution of 28 June 1983

Communication to the Council, concerning a Community Plan of Action relating to the Evaluation of Community Research and
Development Activities for the years 1987 to 1991, COM (86) 660 final, 20 November 1986

CREST advice to Council and the Commission on the monitoring and evaluation procedures for Community research programmes;
adopted at its meeting of 18 May 1995.

Recommendation n° 4 of the Memorandum from the President, Mrs. Gradin and Mr. Liikanen, SEC (95) 1301/4 of 22 July 1995;
and Communication to the Commission on Evaluation (concrete steps towards best practice in the Commission). SEC ...

present framework, reform of financial management culture, and partnership with
Member States. In adopting the second phase of this programme the Commission
introduced, among other things, the requirement for systematic evaluation for all
Community programmes and actions and clarified that it is primarily the responsibility
of operational Directorate Generals to carry out the task of evaluation.

3. EXISTING SITUATION

The legislative Decisions on the fourth Framework Programme and their specific
programmes [6] require the Commission (see Table 1):

For the Framework Programme:

     - to continually and systematically monitor, with appropriate assistance from
independent external experts, the progress in relation to its initial objectives (Articles
4.1); and

     - to have an external assessment conducted by independent qualified experts into the
management of and progress with Community activities carried out during the five
years preceding this assessment and communicate the assessment, accompanied by
the Commission's comments, to the European Parliament, the Council and the
Economic and Social Committee prior to presenting a proposal for the fifth
Framework Programme (Article 4.2);

For the Specific Programmes:

     - to continually and systematically monitor, with appropriate assistance from
independent external experts, the progress within the programme (Article 4.1);

     - to have an external assessment conducted by independent qualified experts of the
activities carried out within the domains covered by the programme and their
management during the five years preceding this assessment (Article 4.2);

     - and, on completion of each specific programme, to provide an independent final
evaluation of the results achieved compared with objectives and to forward this
evaluation to the European Parliament, the Council and the Economic and Social
Committee (Article 4.3).

To implement these legislative requirements with the existing methodology would lead

to:

i) a need to conduct 2 major evaluation exercises (one mid-way through the fourth

Framework Programme and the other on completion), in each case for about 20

programmes;
ii) too short a time interval between these 2 evaluations to be able to detect significant

progress;
iii) final evaluations being required before significant S&T results are achieved and their

impacts become apparent; and
iv) a requirement to clarify the roles of independent external experts who would provide

'appropriate assistance'.

Council Decisions: 94/801/EC or 23 November 1994; 94/572/EC of 27 July 1994; 94/802/EC of 23 November 1994; 94/571/EC
of 27 July 1994; 94/803/EC of 23 November 1994; 94/911/EC ofl5 December 1994; 94/804/EC of 23 November 1994; 94/912/EC
or 15 December 1994; 94/9I3/EC of. 15 December 1994; 94/805/EC of 23 November 1994; 94/806/EC of 23 November 1994;
94/914/ECof 15 December 1994; 94/915/EC of 15 December 1994; 94/807/EC of 23 November 1994: 94/9 IT/EC of 15 December
1994; 94/916/IIC of 15 December 1994; 94/920/1-ura loin of 15 December 1994; 94/799/Euratom of 8 December 1994; and
Council Decisions implemented by the Joint Research Centre: 94/918/EC or 15 December 1994; 94/919/Euratom of 15 December
1994.

4. APPROACH FOR RATIONALIZATION

The following outlines a rationalised approach which the Commission intends to
implement. It takes into account the CREST advice and, where appropriate, the
principles of the SEM 2000 initiative.

4.1 How to rationalize

The CREST advice recognises that the complex evaluation requirements above can in
practice be satisfied by a rationalized approach, while avoiding bureaucracy and
evaluation for its own sake (see Table 1). Programme monitoring and evaluation
requirements will be satisfied by two kinds of actions relating to the Framework
Programme and its specific programmes:

i) continuous monitoring, with appropriate assistance from independent external
experts, involving annual reporting; and
ii) five-year assessment mid-way through implementation. This assessment will be
conducted by independent external experts and will incorporate the final evaluation
under the previous Framework Programme. The final evaluation for any ongoing
specific programmme will be incorporated in the subsequent five-year assessment.
Thus, the final evaluation (Article 4.3) will be independent from the five-year
assessment (Article 4.2) for any one programme.

These two activities, monitoring and evaluation, are distinct and separate exercises.

For the Framework Programme, the five-year assessment, together with the
Commission's comments, will be communicated to the European Parliament, the Council
and the Economic and Social Committee prior to presenting a proposal for the next
Framework Programme (Article 4.2).

Moreover, the CREST advice suggests that the issues to be addressed will include:

the consistency of the selection of projects with the objectives of the specific
programme and the work programme;
the extent to which selected projects or clusters of projects are fulfilling the wider
policy objectives of the Community, in particular in areas of relevance to the
specific programme concerned;
the progress and output of projects against the original targets set;
cases where the independent monitoring teams consider the results will have a
particularly important impact, or where poor performance requires further
examination;

the efficiency and transparency of the management of the programme, including the
development of calls, the assessment and selection process, contract negotiation and
disbursement of funds, as well as internal Commission co-ordination;

the use of special measures and support activities (e.g. to support SMEs, improve
dissemination of information, etc);
the changes that may be needed to the balance of the programme or to the strategy
for implementation in the light of experience and changes in the wider environment.

The CREST advice further suggests a set of input and output performance indicators for
the monitoring and evaluation processes (see Point 5.1 below).

4.2 How to maximize benefit

An integrated monitoring and evaluation approach, with the assistance of external
experts, is outlined below where continuous monitoring reports will in time provide
significant supplementary information to the annual report required under Article 130p
of the Treaty on European Union [7] . The five-year assessment of programmes will provide
a major input to discussions on future Community RTD activities.

In this move to continuous monitoring and five-year assessment, results will be available
at the time when decisions have to be taken and it will be possible to have a global
overview of the state of implementation at regular intervals. In addition, evaluations will
form a 'coherent ensemble'. Lessons from evaluations will be able to be drawn not only
for specific programmes, but also for the Framework Programme and the whole
Community S&T policy.

Appropriate involvement of outside experts and bodies (notably CREST and Programme
Committees), and openness of procedures, will enhance the independence and
transparency of evaluations whilst recognizing the need to respect confidentiality of
certain data. The external experts will discuss their reports with the Programme
Managers and the Programme Committees. This process should facilitate implementation
of the experts' recommendations. In addition, a new level of accountability is introduced
through publishing the Commission's responses to recommendations produced by the
five-year assessment panel of the Framework Programme.

5. EXTERNAL MONITORING

5.1 Specific Programmes

The .primary functions of external monitoring will be to assess the progress of
programmes in order to assist Programme Managers in adapting specific programmes
to developments, and to contribute to the five-year assessment of such programmes.
Monitoring will be a quick response mechanism providing an annual picture of
programme development.

Monitoring will be based on continuous and systematic collection of data by Programme
Managers about the progress and results achieved in the specific programmes. With the
assistance of CREST, a set of suggested performance programme indicators have been
identified which primarily relate to shared-cost programmes under Activity 1 of the
fourth Framework Programme. The other specific programmes, such as the
dissemination and optimization of results and the thermonuclear fusion, follow different
implementation procedures for which different performance indicators may be
appropriate. The Joint Research Centre (JRC) participation in specific programmes will
be considered like any other participant, while the JRC direct-action activities will be
reported on through the "Observations of the Board of Governors on the JRC Annual
Report" which will directly constitute an input to the overall Framework Programme
monitoring exercise.

Treaty on European Union, Article 130p: "At the beginning of each year the Commission shall send a report to the European
Parliament and the Council. The report shall include information on research and technological development activities and the
dissemination of results during the previous year, and the work programme for the current year."

The relevant and available data for each specific programme contained in the
performance indicators will be analyzed by a small panel of independent experts for each
specific programme who would provide an external view.

The monitoring panels will not duplicate the functions of Programme Committees, nor
will they conflict with them, but add an extra dimension, i.e. of independent assessment
concerning the overall performance and achievements of each specific programme. The
monitoring panels will assess programme output against.programme objectives and
against the progress in implementation.

Each panel will present the results of its analysis to the relevant Programme
Management which will take appropriate actions.

5.2 Framework Programme

The Framework Programme monitoring will assess the year-on-year implementation of
the overall programme and will contribute to its five-year assessment. An independent
external experts panel will review the outcome of examinations of the specific
programmes and assess the overall progress in relation to objectives, priorities and
progress in implementation of the entire Framework Programme. In assessing the whole
Framework Programme, the contribution to the objective of strengthening the scientific
and technological bases of Community industry and encouraging competitiveness at
international level, as well as the contribution to the implementation of other Community
policies shall be taken into consideration.

The panel will present the results of its analysis to the Framework Programme
Management which will take appropriate actions.

**6.** **EXTERNAL FIVE-YEAR ASSESSMENTS OF SPECIFIC PROGRAMMES AND**

**FRAMEWORK PROGRAMME**

For specific programmes and the entire Framework Programme, five-year assessments
timed to provide input to discussions on.future legislative decisions on Community RTD
activities, will be produced by panels of independent external experts. The annual
monitoring reports, detailed above, will provide an input to the five-year assessment
which will include a more strategic dimension. (For example, long-term projects which
have been developed in successive Framework Programmes may be assessed over a
longer time scale.) These assessments will be ready when the proposal for the next
Framework Programme is discussed.

The five-year assessments, incorporating the final evaluations under the previous
Framework Programme, will include three principal elements, each looking at distinct,
but inter-related features:

i) assessing relevance, i.e. whether the initial objectives are still valid against
evolving S&T, industrial and socio-economic conditions;
ii) assessing efficiency, i.e. whether the objectives have been pursued in a costeffective manner; and,
iii) assessing effectiveness, i.e. whether the initial objectives have been achieved:
contributions to strengthening the scientific and technological bases and the
competitiveness of Community industry, as well as contributions to all relevant
Community policies.

Special attention will be paid to the development of reliable methodologies for the
evaluation of RTD project results. To this end, an _ad hoc_ Working Group has been
created which includes representatives from industry. It will recommend, with the help
of external contributions, a common methodology to be used for the evaluation of results
of RTD projects, particularly those with industrial relevance.

The five-year assessments might pay particular attention to the coherence between
Community and national S&T policies with a view to enhancing their mutual
consistency [8] .

For the specific programmes, as the five year assessment will incorporate the final
evaluation of the previous specific programme, the assessment reports will be forwarded
to the European Parliament, the Council and the Economic and Social Committee, as
required in Article 4.3 of the relevant Decisions. For the Framework Programme the
five-year assessment report, accompanied by the Commission's comments, will be
forwarded to the same bodies as above prior to presenting a proposal for the next
Framework Programme, as required in Article 4.2 of the relevant Decision.

7. IMPLEMENTATION

The Commission is responsible for the monitoring and evaluation of Community RTD
efforts. Appropriate consultation of CREST, Programme Committees and other relevant
bodies will be sought. Independent external experts will be contracted to undertake the
tasks of monitoring and five-year assessment and preparing reports.

Monitoring panels for the specific programmes will normally comprise 3 experts with
appropriate experience and stature: one from industry, one from academia, and one with
experience of RTD programme evaluation. Monitoring panels for the Framework
Programme will normally comprise 6-7 experts, the larger number reflecting the wider
scope of their work. A similar balance of industrial, academic and programme evaluation
experience will be sought.

For the five-year assessments, the panels, while maintaining a minimum number of
experts, will be slightly larger than for monitoring. The size of the panels for the
specific programme assessments will be determined in relation to the objectives and
scope of the respective programmes. A panel of 6-8 experts will be required for the
Framework Programme assessment. The five-year assessment panels might include a few
experts from the related monitoring panels to ensure continuity between the two different
exercises. The experts will be drawn from a variety of backgrounds, appropriate to the
programme under consideration, and will be expected to have extensive experience in
order to fully examine horizontal issues.

Special attention will be paid to ensure the coherence of monitoring and five-year
assessment and to maintain the highest possible degree of transparency for the exercise.

Regular reporting will be made once a year, correlated with the publication of the
Annual Report requested under Article 130p of the Treaty on European Union.

Treaty on European Union, Article I30h: "The Community and the Member States shall co-ordinate their research and technological

development activities so as to ensure that national policies and Community policy are mutually consistent".

**8.** **TIMING**

**Every effort will be made to implement this scheme as quickly as possible.**

**The** **1995** **monitoring exercise has been completed. The external** **experts'** **reports on the**
**specific programmes and on the overall Framework Programme are available.**

**In respect of the forthcoming five-year assessment, the first panels will be expected to**
**start their work during Spring 1996 with specific programme reports expected by**
**Summer/Autumn** **1996 and the report on the overall Framework Programme before the**
**end of the year.**

**The rationalized approach, suggested above, will be reviewed after three years of**
**implementation.**

**9.** **CONCLUSIONS**

**The series of regular annual monitoring reports and five-year assessment reports will**
**provide a comprehensive overview of the management of programmes (Framework**
**Programme and specific programmes), their implementation and benefits derived. The**
**involvement of external experts and the openness of procedures will enhance the**
**independence and transparency of** **monitoring** **and evaluation. The publication of**
**Commission's responses** **to** **recommendations** **produced by** **the five-year assessment panel**
**of the Framework Programme will introduce a new level of** **accountability.** **Meetings**
**between the experts and Programme Managers and the Programme Committees on the**
**results of monitoring and evaluation will ensure mutual understanding and facilitate the**
**implementation of appropriate recommendations. All these features will further enhance**
**the credibility of Community RTD evaluation efforts.**

**The five-year assessment reports will be given to the European Parliament, the Council**
**and the Economic and Social Committee thus providing input to future legislative**
**decisions on Community RTD activities.**

**Through this Communication, the Commission invites the Council and the European**
**Parliament to take note of this rationalized approach which takes account of the current**
**legislative requirements for monitoring and evaluation.**

TABLE 1: EXISTING SITUATION vs. SUGGESTED SCHEME

```
1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 Y E A R

       FP 3 : 1990-1994

```

FP 4 : 1994-1998

1. LEGISLATIVE OBLIGATIONS:

FP3

Review

Specific programmes

FP 5 : 1998-2002

Final

evaluation

Continuous monitoring

, Continuous monitoring,

reporting at the beginning

of each year (*)

**!**

FP3

Final

evaluation

Final

evaluation

5 year

assessment
###### **h-L**

State of 5 year
implementation I assessment
# **n**

**(*>** **(*)** **«** **(*>**

**2)** FP4

2. SUGGESTED SCHEME:

### **{**

State of

implementation

Specific programmes f

###### FP4 {

5-year assessment,
(incl. final evaluation of FP3)
approx. period covered: 1991-1995

5-year assessment
(incl. final evaluation of FP4)
approx. period covered: 1995-1999

JFP3 + FP4 | ( + FP5 + ...'.) { Specificprogramine, j

**(*)** **(*)** **(*)** **(*)** **(*)** **(*)**

Note : arrows indicate the approximate periods during which evaluation reports should be issued.

Continuous monitoring,
reporting at the beginning

of each year (*)

**ISSN 0254-1475**

#### **COM(96) 220 final**

## **DOCUMENTS**

##### **EN 15** **Catalogue number : CB-CO-96-231-EN-C** **ISBN 92-78-04209-9**

**Office for Official Publications of** **the** **European Communities**

**L-2985** **Luxembourg**