\s 
COMMISSION OF THE EUROPEAN COMMUNITIES 
Brussels, 22.05.1996 
COM(96)220 final 
COMMUNICATION FROM THE COMMISSION 
TO THE COUNCIL AND THE EUROPEAN PARLIAMENT 
INDEPENDENT EXTERNAL MONITORING AND EVALUATION OF 
COMMUNITY ACTIVITIES IN THE AREA OF RESEARCH AND 
TECHNOLOGICAL DEVELOPMENT 

SUMMARY 
The Commission is continually seeking to demonstrate and enhance the credibility and 
efficiency of its efforts in relation to evaluation of Community RTD actions. In the light of 
the recent legislative Decisions on the fourth Framework Programme and the related specific 
programmes, which prescribe a complex evaluation scheme and an active role for independent 
external experts in this scheme, the Commission decided to take a fresh look at Community 
RTD evaluation efforts. The Scientific and Technical Research Committee (CREST) was 
consulted for advice; subsequently, CREST established an Evaluation Sub-Committee. The 
Council's Research Group and Atomic Questions Group as well as the European Parliament's 
Committee on Research, Technological Development and Energy (CERT) have been informed 
of this development. 
This Communication summarizes the evaluation required by legislative Decisions and outlines 
a rationalized approach, taking into account the CREST advice. The approach will produce 
results in a timely, efficient and cost-effective manner. It concentrates on the continuous 
monitoring of specific programmes and the Framework Programme and their five-year 
assessment. Regular reporting will be made once a year, correlated with the publication of 
the Annual Report requested under Article 130p of the Treaty on European Union. The 
approach has been introduced on an experimental basis in 1995 and will be reviewed after 
three years of implementation. 
The approach builds upon the solid foundation of experience already acquired by the 
Commission from past evaluations and takes into account a recent initiative on concrete steps 
towards best evaluation practice in the Commission. The main feature of the approach is a 
further development towards coherent monitoring and evaluation of Community RTD actions. 
In particular, the approach: 
(i) introduces continuous monitoring and develops through programme evaluation to S&T 
policy evaluation; 
(ii) spells out the implementation principles; 
(iii) involves appropriate assistance from independent external experts; and 
(iv) should produce results in time for discussion on future programme Decisions. 
The Commission is responsible for the monitoring and evaluation of Community RTD efforts. 
However, the involvement of outside experts and bodies and the openness of procedures will 
enhance the independence and transparency of monitoring and evaluation. In addition, a new 
level of accountability is introduced through publication of the Commission's responses to 
recommendations produced by the five-year assessment panel of the Framework Programme. 
This will all enhance the credibility of Community RTD evaluation. 
Through this Communication, the Commission invites the Council and the European 
Parliament to take note of this rationalized approach which takes account of the current 
legislative requirements for monitoring and evaluation. 

1. INTRODUCTION 
The Commission's White Paper "Growth, Competitiveness, Employment", COM(93)700 
final, identifies increased RTD effort and its adaptation to new market conditions as one 
of the key means to improve the industrial and technological performance of Europe. 
Evaluation - both the evaluation of the programmes (their management, monitoring 
quality, efficiency of internal procedures, etc.) and the evaluation of results (increase in 
scientific and technical knowledge, dissemination and optimization of results, industrial 
exploitation of RTD results and their relation to innovation processes, measurement of 
long-term economic and social effects, etc.) - plays an important role in this regard by 
contributing to: the efficient implementation of RTD efforts; coherence with all relevant 
policies; and best use of public resources. 
The legislative Decisions on the fourth Framework Programme1 and its specific 
programmes, require the Commission to implement a complex evaluation scheme (see 
Point 3 below). In this Communication the Commission presents a rationalized approach 
which will produce results in a timely, efficient and cost-effective manner. 
2. EVOLUTION OF COMMUNITY RTD EVALUATION EFFORTS 
The systematic evaluation of Community RTD activities was confirmed with the Plan of 
Action of 19832 and has since evolved further3. The use of external experts has been a 
key feature in promoting the independence of evaluation. 
In 1993, the Council and the European Parliament invited the Commission to continue 
to improve the credibility of the evaluation of Community RTD programmes while 
ensuring its independence. Subsequently, the legislative Decisions on the fourth 
Framework Programme included additional requirements concerning evaluation. 
In 1994, the Commission Services involved in the implementation of the Framework 
Programme, prepared a reflection document as a first step in response to the Council and 
the European Parliament in this regard. The document was transmitted to CREST, CERT 
and the European Science and Technology Assembly (ESTA). 
In 1995, CREST produced advice4 which included the establishment of a CREST 
Evaluation Sub-Committee. The Research Group and the Atomic Questions Group 
(Council), as well as CERT (Parliament), have been kept informed of this development. 
On 21 March 1995 the Commission approved a programme5 for "Sound and Efficient 
Financial Management (SEM 2000)", divided into three phases: consolidation within the 
In this document, the term "fourth Framework Programme" comprises Community activities resulting from two Decisions: J^ Decision N° 
1110/94/EEC concerning the 4th Framework Programme of the European Community activities in the field of research and technological 
development and demonstration (1994-1998); and, 2L Decision N° 94/268/Euratom concerning a framework programme of Community 
activities in the field of research and training for the European Atomic Energy Community (1994-1998). 
Council Resolution of 28 June 1983 
Communication to the Council, concerning a Community Plan of Action relating to the Evaluation of Community Research and 
Development Activities for the years 1987 to 1991, COM (86) 660 final, 20 November 1986 
CREST advice to Council and the Commission on the monitoring and evaluation procedures for Community research programmes; 
adopted at its meeting of 18 May 1995. 
Recommendation n° 4 of the Memorandum from the President, Mrs. Gradin and Mr. Liikanen, SEC (95) 1301/4 of 22 July 1995; 
and Communication to the Commission on Evaluation (concrete steps towards best practice in the Commission). SEC ... 
present framework, reform of financial management culture, and partnership with 
Member States. In adopting the second phase of this programme the Commission 
introduced, among other things, the requirement for systematic evaluation for all 
Community programmes and actions and clarified that it is primarily the responsibility 
of operational Directorate Generals to carry out the task of evaluation. 
3. EXISTING SITUATION 
The legislative Decisions on the fourth Framework Programme and their specific 
programmes6 require the Commission (see Table 1): 
For the Framework Programme: 
- to continually and systematically monitor, with appropriate assistance from 
independent external experts, the progress in relation to its initial objectives (Articles 
4.1); and 
- to have an external assessment conducted by independent qualified experts into the 
management of and progress with Community activities carried out during the five 
years preceding this assessment and communicate the assessment, accompanied by 
the Commission's comments, to the European Parliament, the Council and the 
Economic and Social Committee prior to presenting a proposal for the fifth 
Framework Programme (Article 4.2); 
For the Specific Programmes: 
- to continually and systematically monitor, with appropriate assistance from 
independent external experts, the progress within the programme (Article 4.1); 
- to have an external assessment conducted by independent qualified experts of the 
activities carried out within the domains covered by the programme and their 
management during the five years preceding this assessment (Article 4.2); 
- and, on completion of each specific programme, to provide an independent final 
evaluation of the results achieved compared with objectives and to forward this 
evaluation to the European Parliament, the Council and the Economic and Social 
Committee (Article 4.3). 
To implement these legislative requirements with the existing methodology would lead 
to: 
i) a need to conduct 2 major evaluation exercises (one mid-way through the fourth 
Framework Programme and the other on completion), in each case for about 20 
programmes; 
ii) too short a time interval between these 2 evaluations to be able to detect significant 
progress; 
iii) final evaluations being required before significant S&T results are achieved and their 
impacts become apparent; and 
iv) a requirement to clarify the roles of independent external experts who would provide 
'appropriate assistance'. 
Council Decisions: 94/801/EC or 23 November 1994; 94/572/EC of 27 July 1994; 94/802/EC of 23 November 1994; 94/571/EC 
of 27 July 1994; 94/803/EC of 23 November 1994; 94/911/EC ofl5 December 1994; 94/804/EC of 23 November 1994; 94/912/EC 
or 15 December 1994; 94/9I3/EC of. 15 December 1994; 94/805/EC of 23 November 1994; 94/806/EC of 23 November 1994; 
94/914/ECof 15 December 1994; 94/915/EC of 15 December 1994; 94/807/EC of 23 November 1994: 94/9 IT/EC of 15 December 
1994; 94/916/IIC of 15 December 1994; 94/920/1-ura loin of 15 December 1994; 94/799/Euratom of 8 December 1994; and 
Council Decisions implemented by the Joint Research Centre: 94/918/EC or 15 December 1994; 94/919/Euratom of 15 December 
1994. 
4. APPROACH FOR RATIONALIZATION 
The following outlines a rationalised approach which the Commission intends to 
implement. It takes into account the CREST advice and, where appropriate, the 
principles of the SEM 2000 initiative. 
4.1 How to rationalize 
The CREST advice recognises that the complex evaluation requirements above can in 
practice be satisfied by a rationalized approach, while avoiding bureaucracy and 
evaluation for its own sake (see Table 1). Programme monitoring and evaluation 
requirements will be satisfied by two kinds of actions relating to the Framework 
Programme and its specific programmes: 
i) continuous monitoring, with appropriate assistance from independent external 
experts, involving annual reporting; and 
ii) five-year assessment mid-way through implementation. This assessment will be 
conducted by independent external experts and will incorporate the final evaluation 
under the previous Framework Programme. The final evaluation for any ongoing 
specific programmme will be incorporated in the subsequent five-year assessment. 
Thus, the final evaluation (Article 4.3) will be independent from the five-year 
assessment (Article 4.2) for any one programme. 
These two activities, monitoring and evaluation, are distinct and separate exercises. 
For the Framework Programme, the five-year assessment, together with the 
Commission's comments, will be communicated to the European Parliament, the Council 
and the Economic and Social Committee prior to presenting a proposal for the next 
Framework Programme (Article 4.2). 
Moreover, the CREST advice suggests that the issues to be addressed will include: 
the consistency of the selection of projects with the objectives of the specific 
programme and the work programme; 
the extent to which selected projects or clusters of projects are fulfilling the wider 
policy objectives of the Community, in particular in areas of relevance to the 
specific programme concerned; 
the progress and output of projects against the original targets set; 
cases where the independent monitoring teams consider the results will have a 
particularly important impact, or where poor performance requires further 
examination; 
the efficiency and transparency of the management of the programme, including the 
development of calls, the assessment and selection process, contract negotiation and 
disbursement of funds, as well as internal Commission co-ordination; 
the use of special measures and support activities (e.g. to support SMEs, improve 
dissemination of information, etc); 
the changes that may be needed to the balance of the programme or to the strategy 
for implementation in the light of experience and changes in the wider environment. 
The CREST advice further suggests a set of input and output performance indicators for 
the monitoring and evaluation processes (see Point 5.1 below). 
4.2 How to maximize benefit 
An integrated monitoring and evaluation approach, with the assistance of external 
experts, is outlined below where continuous monitoring reports will in time provide 
significant supplementary information to the annual report required under Article 130p 
of the Treaty on European Union7. The five-year assessment of programmes will provide 
a major input to discussions on future Community RTD activities. 
In this move to continuous monitoring and five-year assessment, results will be available 
at the time when decisions have to be taken and it will be possible to have a global 
overview of the state of implementation at regular intervals. In addition, evaluations will 
form a 'coherent ensemble'. Lessons from evaluations will be able to be drawn not only 
for specific programmes, but also for the Framework Programme and the whole 
Community S&T policy. 
Appropriate involvement of outside experts and bodies (notably CREST and Programme 
Committees), and openness of procedures, will enhance the independence and 
transparency of evaluations whilst recognizing the need to respect confidentiality of 
certain data. The external experts will discuss their reports with the Programme 
Managers and the Programme Committees. This process should facilitate implementation 
of the experts' recommendations. In addition, a new level of accountability is introduced 
through publishing the Commission's responses to recommendations produced by the 
five-year assessment panel of the Framework Programme. 
5. EXTERNAL MONITORING 
5.1 Specific Programmes 
The .primary functions of external monitoring will be to assess the progress of 
programmes in order to assist Programme Managers in adapting specific programmes 
to developments, and to contribute to the five-year assessment of such programmes. 
Monitoring will be a quick response mechanism providing an annual picture of 
programme development. 
Monitoring will be based on continuous and systematic collection of data by Programme 
Managers about the progress and results achieved in the specific programmes. With the 
assistance of CREST, a set of suggested performance programme indicators have been 
identified which primarily relate to shared-cost programmes under Activity 1 of the 
fourth Framework Programme. The other specific programmes, such as the 
dissemination and optimization of results and the thermonuclear fusion, follow different 
implementation procedures for which different performance indicators may be 
appropriate. The Joint Research Centre (JRC) participation in specific programmes will 
be considered like any other participant, while the JRC direct-action activities will be 
reported on through the "Observations of the Board of Governors on the JRC Annual 
Report" which will directly constitute an input to the overall Framework Programme 
monitoring exercise. 
Treaty on European Union, Article 130p: "At the beginning of each year the Commission shall send a report to the European 
Parliament and the Council. The report shall include information on research and technological development activities and the 
dissemination of results during the previous year, and the work programme for the current year." 
The relevant and available data for each specific programme contained in the 
performance indicators will be analyzed by a small panel of independent experts for each 
specific programme who would provide an external view. 
The monitoring panels will not duplicate the functions of Programme Committees, nor 
will they conflict with them, but add an extra dimension, i.e. of independent assessment 
concerning the overall performance and achievements of each specific programme. The 
monitoring panels will assess programme output against.programme objectives and 
against the progress in implementation. 
Each panel will present the results of its analysis to the relevant Programme 
Management which will take appropriate actions. 
5.2 Framework Programme 
The Framework Programme monitoring will assess the year-on-year implementation of 
the overall programme and will contribute to its five-year assessment. An independent 
external experts panel will review the outcome of examinations of the specific 
programmes and assess the overall progress in relation to objectives, priorities and 
progress in implementation of the entire Framework Programme. In assessing the whole 
Framework Programme, the contribution to the objective of strengthening the scientific 
and technological bases of Community industry and encouraging competitiveness at 
international level, as well as the contribution to the implementation of other Community 
policies shall be taken into consideration. 
The panel will present the results of its analysis to the Framework Programme 
Management which will take appropriate actions. 
6. EXTERNAL FIVE-YEAR ASSESSMENTS OF SPECIFIC PROGRAMMES AND 
FRAMEWORK PROGRAMME 
For specific programmes and the entire Framework Programme, five-year assessments 
timed to provide input to discussions on.future legislative decisions on Community RTD 
activities, will be produced by panels of independent external experts. The annual 
monitoring reports, detailed above, will provide an input to the five-year assessment 
which will include a more strategic dimension. (For example, long-term projects which 
have been developed in successive Framework Programmes may be assessed over a 
longer time scale.) These assessments will be ready when the proposal for the next 
Framework Programme is discussed. 
The five-year assessments, incorporating the final evaluations under the previous 
Framework Programme, will include three principal elements, each looking at distinct, 
but inter-related features: 
i) assessing relevance, i.e. whether the initial objectives are still valid against 
evolving S&T, industrial and socio-economic conditions; 
ii) assessing efficiency, i.e. whether the objectives have been pursued in a cost-
effective manner; and, 
iii) assessing effectiveness, i.e. whether the initial objectives have been achieved: 
contributions to strengthening the scientific and technological bases and the 
competitiveness of Community industry, as well as contributions to all relevant 
Community policies. 
Special attention will be paid to the development of reliable methodologies for the 
evaluation of RTD project results. To this end, an ad hoc Working Group has been 
created which includes representatives from industry. It will recommend, with the help 
of external contributions, a common methodology to be used for the evaluation of results 
of RTD projects, particularly those with industrial relevance. 
The five-year assessments might pay particular attention to the coherence between 
Community and national S&T policies with a view to enhancing their mutual 
consistency8. 
For the specific programmes, as the five year assessment will incorporate the final 
evaluation of the previous specific programme, the assessment reports will be forwarded 
to the European Parliament, the Council and the Economic and Social Committee, as 
required in Article 4.3 of the relevant Decisions. For the Framework Programme the 
five-year assessment report, accompanied by the Commission's comments, will be 
forwarded to the same bodies as above prior to presenting a proposal for the next 
Framework Programme, as required in Article 4.2 of the relevant Decision. 
7. IMPLEMENTATION 
The Commission is responsible for the monitoring and evaluation of Community RTD 
efforts. Appropriate consultation of CREST, Programme Committees and other relevant 
bodies will be sought. Independent external experts will be contracted to undertake the 
tasks of monitoring and five-year assessment and preparing reports. 
Monitoring panels for the specific programmes will normally comprise 3 experts with 
appropriate experience and stature: one from industry, one from academia, and one with 
experience of RTD programme evaluation. Monitoring panels for the Framework 
Programme will normally comprise 6-7 experts, the larger number reflecting the wider 
scope of their work. A similar balance of industrial, academic and programme evaluation 
experience will be sought. 
For the five-year assessments, the panels, while maintaining a minimum number of 
experts, will be slightly larger than for monitoring. The size of the panels for the 
specific programme assessments will be determined in relation to the objectives and 
scope of the respective programmes. A panel of 6-8 experts will be required for the 
Framework Programme assessment. The five-year assessment panels might include a few 
experts from the related monitoring panels to ensure continuity between the two different 
exercises. The experts will be drawn from a variety of backgrounds, appropriate to the 
programme under consideration, and will be expected to have extensive experience in 
order to fully examine horizontal issues. 
Special attention will be paid to ensure the coherence of monitoring and five-year 
assessment and to maintain the highest possible degree of transparency for the exercise. 
Regular reporting will be made once a year, correlated with the publication of the 
Annual Report requested under Article 130p of the Treaty on European Union. 
Treaty on European Union, Article I30h: "The Community and the Member States shall co-ordinate their research and technological 
development activities so as to ensure that national policies and Community policy are mutually consistent". 
8. TIMING 
Every effort will be made to implement this scheme as quickly as possible. 
The 1995 monitoring exercise has been completed. The external experts' reports on the 
specific programmes and on the overall Framework Programme are available. 
In respect of the forthcoming five-year assessment, the first panels will be expected to 
start their work during Spring 1996 with specific programme reports expected by 
Summer/Autumn 1996 and the report on the overall Framework Programme before the 
end of the year. 
The rationalized approach, suggested above, will be reviewed after three years of 
implementation. 
9. CONCLUSIONS 
The series of regular annual monitoring reports and five-year assessment reports will 
provide a comprehensive overview of the management of programmes (Framework 
Programme and specific programmes), their implementation and benefits derived. The 
involvement of external experts and the openness of procedures will enhance the 
independence and transparency of monitoring and evaluation. The publication of 
Commission's responses to recommendations produced by the five-year assessment panel 
of the Framework Programme will introduce a new level of accountability. Meetings 
between the experts and Programme Managers and the Programme Committees on the 
results of monitoring and evaluation will ensure mutual understanding and facilitate the 
implementation of appropriate recommendations. All these features will further enhance 
the credibility of Community RTD evaluation efforts. 
The five-year assessment reports will be given to the European Parliament, the Council 
and the Economic and Social Committee thus providing input to future legislative 
decisions on Community RTD activities. 
Through this Communication, the Commission invites the Council and the European 
Parliament to take note of this rationalized approach which takes account of the current 
legislative requirements for monitoring and evaluation. 

TABLE 1: EXISTING SITUATION vs. SUGGESTED SCHEME 
1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 YEAR 
FP 3 : 1990-1994 
1. LEGISLATIVE OBLIGATIONS: 
FP3 
! Specific programmes 
FP3 
2) FP4 Review 
State of 
implementation 
{ Specific programmes f 
{ FP4 FP 4 : 1994-1998 
Final 
evaluation 
Final 
evaluation 
5 year 
assessment 
h-L 
State of 5 year 
implementation I assessment n 
(*> (*) « (*> FP 5 : 1998-2002 
Final 
evaluation 
Continuous monitoring 
, Continuous monitoring, 
reporting at the beginning 
of each year (*) 
2. SUGGESTED SCHEME: 
5-year assessment, 
(incl. final evaluation of FP3) 
approx. period covered: 1991-1995 
JFP3 + FP4 | (+ FP5 + ...'.) { Specificprogramine, j 5-year assessment 
(incl. final evaluation of FP4) 
approx. period covered: 1995-1999 
(*) (*) (*) (*) (*) (*) Continuous monitoring, 
reporting at the beginning 
of each year (*) 
Note : arrows indicate the approximate periods during which evaluation reports should be issued. 


ISSN 0254-1475 
COM(96) 220 final 
DOCUMENTS 
EN 15 
Catalogue number : CB-CO-96-231-EN-C 
ISBN 92-78-04209-9 
Office for Official Publications of the European Communities 
L-2985 Luxembourg