COMMISSION OF THE EUROPEAN COMMUNITIES 
Brussels, 16.04.1997 
COM(97) 149 final 
COMMUNICATION FROM THE COMMISSION 
EU RJESEARCH AND TECHNOLOGICAL 
DEVELOPMENT ACTIVITIES: 
COMMISSION'S RESPONSES TO THE RECOMMENDATIONS 
OF THE 
INDEPENDENT EXTERNAL ASSESSMENTS 
IN THE DOMAINS COVERED 
BY 
THE SPECIFIC PROGRAMMES AND THE JRC INSTITUTES 
UNDER THE FOURTH FRAMEWORK PROGRAMME 
AND 
EURATOM FRAMEWORK PROGRAMME 

COMMISSION OF THE EUROPEAN COMMUNITIES 
Brussels, 16.04.1997 
COM(97) 149 final 
COMMUNICATION FROM THE COMMISSION 
EU RESEARCH AND TECHNOLOGICAL 
DEVELOPMENT ACTIVITIES: 
COMMISSIONS RESPONSES TO THE RECOMMENDATIONS 
OF THE 
INDEPENDENT EXTERNAL ASSESSMENTS 
OF THE LAST FIVE YEARS OF ACTIVITIES 
IN THE DOMAINS COVERED 
BY 
THE SPECIFIC PROGRAMMES AND THE JRC INSTITUTES 
UNDER THE FOURTH FRAMEWORK PROGRAMME 
AND 
EURATOM FRAMEWORK PROGRAMME 

INTRODUCTION 
The Commission welcomes the reports resulting from the 5-year assessment exercises carried out by 
high level, independent expert panels. The Commission appreciates the panels' efforts to reach 
balanced conclusions and valuable recommendations. 
The Council Decision for each Specific Programme1 requires "an external assessment conducted by 
independent experts of the activities carried out within the domains covered by the programme and 
their management during the 5 years preceding this assessment". To this end, a new rationalized 
evaluation approach has been applied which complies with legislative requirements, takes into 
account advice provided by CREST3, and follows the recommendations of the Commission initiative 
(SEM2000) for concrete steps towards best evaluation practices across the Commission's Services4. 
During the second half of 1996, 18 expert panels have assessed RTD efforts sponsored by the EU 
over the previous 5 years under the various Specific Programmes. In addition, 7 independent 
visiting groups were established to assess the Joint Research Centre (JRC) activities (i.e. JRC 
participation in Specific Programmes and its own research programmes) and another high level 
expert to provide an overall evaluation and synthesis of the visiting groups reports. One further 
group was charged with providing an assessment covering all Community RTD activities over the 
last 5 years (EC and Euratom Framework Programmes). All in all, some 170 senior independent 
experts have been involved in the exercise, which subsumes the final evaluations of the programmes 
under the Third Framework Programme, producing 18 Specific Programme reports, 7 JRC 
Institutes reports and one global JRC report, and one overall Framework Programme report. 
This Communication presents the Commission's responses to the major recommendations provided 
in the Specific Programme reports. These recommendations are included in the executive summaries 
which are reproduced in this document for completeness. The opinions expressed in the executive 
summaries are those of the external expert panels and are given under their responsibility. (Each 
Specific Programme is presented in a separate section including three parts: I. Executive Summary; 
II. Commission's Response; and III. Evaluation Panel. The section on JRC is composed of the 
overall evaluation and synthesis report and the list of experts having conducted the assessments of 
the seven Institutes.) 
The Commission notes in particular the comments made by the expert panels concerning the 
management and the procedures of RTD programmes and will, where appropriate, propose 
measures. 
The full expert reports for the Specific Programmes will be published separately. The results of the 
JRC assessments together with the comments of the Board of Governors will be communicated 
separately. 
The Framework Programmes overall assessment together with the Commission's responses will also 
be communicated separately. 
1 Council Decisions: 94/801/EC of 23 November 1994; 94/572/EC of 27 July 1994; 94/802/EC of 23 November 1994; 94/571/EC of 27 
July 1994; 94/803/EC of 23 November 1994; 94/911/EC of 15 December 1994; 94/804/EC of 23 November 1994; 94/912/EC of 15 
December 1994; 94/913/EC of 15 December 1994; 94/805/EC of 23 November 1994; 94/806/EC of 23 November 1994; 94/914/EC of 15 
December 1994; 94/915/EC of 15 December 1994; 94/807/EC of 23 November 1994; 94/917/EC of 15 December 1994; 94/916/EC of 15 
December 1994; 94/920/Euratom of 15 December 1994; 94/799/Euratom of 8 December 1994; and Council Decisions implemented by the 
Joint Research Centre: 94/918/EC of 15 December 1994; 94/919/Euratom of 15 December 1994. 
2 COM(96)22() final, 22 May 1996 
3 
CREST advice to Council and the Commission on the monitoring and evaluation procedures for Community research programmes. 
4 
Communication from Commissioners Liikanen and Gradin in agreement with the President, SEC(96)659/5, 29 April 1996. 
' For the JRC. the relevant articles of the Decisions do not require a response by the Commission to the experts' reports. 

TABLE OF CONTENTS 
page 
1. Telematics application 7 
2. Advanced communication telechnologies and services 19 
3. Information technologies 27 
4. Industrial and materials technologies 43 
5. Measurements and testing 55 
6. Environment and climate 61 
7. Marine science and technologies 73 
8. Biotechnology 81 
9. Biomedicine and health 91 
10. Agriculture and fisheries, including agro-industry 103 
11. Non-nuclear energy Ill 
12. Transport 119 
13. Targeted socio-economic research 125 
14. Cooperation with third countries 135 
15. Dissemination and optimization of results (Innovation) 147 
16. Stimulation of training and mobility 155 
17. Nuclear fission safety 161 
18. Controlled thermonuclear fusion 169 
19. Joint Research Center (JRC) 177 
Annexes: Terms of reference of the five-year assessment 191 
• Annex 1: Specific Programmes 192 
• Annex 2: JRC Institutes 193 

1996 FIVE-YEAR ASSESSMENT 
TELEMATICS APPLICATION 

I. EXECUTIVE SUMMARY 
Key Conclusions 
1. The ICT Programme Structure. A single Information and Communication Technologies 
Programme is recommended, with a major section of it devoted to Applications Systems. This 
would continue on from the Telematics Applications Programme, retaining the successful features of 
that programme, notably strong user involvement in real systems development and validation. 
2. Vision and Strategic Thinking. The Programme needs to develop stronger vision and strategic 
thinking to concentrate the limited resources where development work will be most effective. 
Links with the Policy parts of the Commission need to be strengthened. 
3. The Applications Systems Section Structure. The worlds of multi-media telematics and networks 
have developed considerably since the Telematics Programme was authorised in June 1991, and are 
continuing to evolve very fast. The transition from the 4th Framework Programme (FP) to the 5th FP 
presents the opportunity to rethink the applications section of the programme in the light of these 
technological, market and socio-political developments. The applications projects should be 
concentrated on the emerging sectors of new telematics use and on those where there is a 
particularly strong economic or social case, such as those thrown up by new developments and 
those related to the information providers part of the "Information Society". 
4. The Application Systems Section Content. In addition to the calls for applications projects 
organised on a Sector basis, a separate series of calls should aim to create integrated systems with 
joint usage of two or more different applications, breaking down the Sector boundaries. In order 
to magnify the impact of the programme, a very important part of the work of all Sectors of the 
Applications programme should be to build up the physical and intellectual infrastructure, 
including standards work, and to create platforms that will help to replicate the individual 
applications projects. Such work should be given priority throughout the Programme. 
5. Administration, etc. The "bureaucracy" surrounding this, and other Framework Programmes has 
steadily increased to the point where key firms and public user bodies are refusing to participate, and 
where it is irresponsible to encourage the participation of the high growth potential, small firms, and 
the new applicants we would most wish to see in the programme. Action must be taken to improve 
delegation and reduce nugatory paper-work. Participants in the Programme find their project work 
inhibited by the endless demands on their time. For example, measures and projects at both Programme 
and Sector level, designed to support the Programme and projects as a whole, seem to the participants 
to have risen considerably in the 4th FP, creating an unfortunate impact in the time and effort they 
cause to the project teams, which is unlikely to be balanced by the benefits. These overhead projects 
should be pruned and normally established only at the request of participants in the applications 
projects. 
Introduction 
1. Remit of the Telematics Applications Programme Five Year Assessment Panel 
The Telematics Five-Year Assessment Panel was formed early in 1996 to cover the RTD activities carried 
out in the field of "telematics" (that is in the field of applications of computer and/or communication 
technologies) during the period from 1991 to early 1996. So the remit of the Panel subsumes the final 
review of the Telematics Programme under the 3rd FP (Framework Programme), and the mid-term review 
under the 4th FP, into this 5-year assessment. There were 13 members of the Review Panel because of the 
complexity and wide coverage of this Telematics Applications Programme. The Panel interviewed the 
senior staff and others connected with the running of the programme, and various individual project 
participants. Working in sub-groups, Panel members examined the work of the individual sectors. 
1. 
Questionnaires were sent to all the main participants in the 3rd FP projects (1860) and to the co-ordinators 
of such of the 4th FP projects as had been authorised at that time (370). 
2. Telematics Applications Programme: History 
The Telematics Applications Programme was preceded by the three exploratory actions, established in 
1989 as part of the 2nd FP. The Programme was established as part of the 3rd FP, covering 7 Sectors, 
with a budget of 430 Mecu. Four other Sectors were added in the 4th FP, with a total Programme budget 
of 912 Mecu. 312 projects were funded under the 3rd FP; so far 422 projects have been funded under the 
4th FP. 
3. Structure of the Full Report 
The Full Report of the Panel consists of the Key Conclusions, as given above in this shortened version; an 
Introduction; an Executive Summary in the form of a Discussion of the Panels Findings; a section 
presenting and explaining the Panels' main Conclusions and Recommendations; and a set of the 
Conclusions and Recommendations on each of the 12 Sectors of the Programme, with the full Sector 
reports given in an Annex. Other Annexes cover the analysis of the questionnaires and finance and project 
statistics. This Shortened Report is based on the Discussion section of the full report; the reader is strongly 
encouraged to look at the full report to do justice to the conclusions, and to the recommendations 
stemming from those conclusions which space prevents setting out in this Shortened Report. 
Discussion of the Key Findings 
1. The Achievements 
The essence of the Telematics Applications Programme lies in Applications; the applications of 
information and communications technology to the problems of explicit categories of users, making use of 
systems that harness emerging technologies of networking and multimedia to provide solutions. There has 
been a significant shift between the 3rd and the 4th FP in the extent and nature of the user involvement in 
the programme. True end users can validate the system approach and the human interfaces, and 
intermediate users or customer bodies can control or at least influence the subsequent deployment of the 
fruits of the R&D work. The Panel found that the majority of the projects in the Programme are firmly 
directed to the solution of perceived problems. Especially in the 4th FP, users are strongly involved in 
drawing up the workplans, in evaluation of the applications, and in the conduct of the approved projects. 
This is much to be welcomed. The so-called "Ten Commandments" of the Programme have proved to be 
an excellent way of encouraging highly desirable trends throughout the Programme, and especially in the 
encouragement of user involvement. It is too early to have proof that it increases the likelihood of project 
success. However, it is a reasonable expectation that true involvement of users in the programme will 
improve the deployment of telematics applications. 
It has been the great achievement of earlier generations of Commission programmes, like ESPRIT and 
RACE, that the academic and industrial research workers of Europe have learnt to co-operate together 
through exposure to each other in the work of the programmes. Now the Telematics Applications 
Programme is pioneering the coming together of the users of Europe. This may be the great contribution 
of the Telematics Applications Programme. Greater coherence in the users, the customers of ICT in 
Europe, will make for faster reacting and better informed markets, to the benefit of all concerned. 
2. The Changed Environment 
The Telematics Applications Programme has been running for some 5 years. During the last few years the 
environment for telematics applications has changed dramatically with the use of the Internet growing 
exponentially, and awareness of the potential benefits of telematics becoming widespread. The use of 
multimedia has become common-place. Prices of computer hardware and the cost of bandwidth are falling 
dramatically, and are predicted to continue to do so. Of course the penetration of multi-media telematics 
10 
1. 
has still got a long way to go. The challenge now is both to accelerate the efficient use of such systems, 
especially where little penetration has yet been achieved, and to encourage the emergence of integrated 
applications. 
3. Time for a Rethink 
In the light of the dramatic changes in the market place, the Telematics Applications Programme 
sometimes displays a certain tiredness. While not throwing away the distinctive and beneficial features of a 
programme that concentrates on real applications with strong user involvement, the limited resources 
must be used in a way that maximises positive impact, and concentrates on the areas of most need 
and where the time-scale of penetration can be best accelerated. In what is now a global market place, 
only quality will lead to success so projects must involve the European bodies most capable of 
producing quality work. Some restructuring and re-engineering of the Programme will be required, 
with the addition to the current main thrust, using applications projects, of new approaches that aim 
to accelerate the take-up and integration of telematics, and its migration towards multimedia. 
3.1 The Opportunity for Europe. The emerging multimedia industries will include some of the largest 
business opportunities of the next two decades. Yet, there is no clear picture of how this futuristic industry 
will evolve. In the presence of such uncertainty, players who articulate a consistent strategy to exploit the 
opportunities and address the risks will shape the evolution of the industry structure. Since Europe 
currently lags well behind the market penetration figures for the United States of America, we have an 
unique opportunity to develop here a strong, early market for these new applications - upon which world 
wide market success can be built. The Applications Programme must be rethought to take into account 
this opportunity. 
3.2 Infrastructure, Standards and Platforms. It is important to ensure that the necessary infrastructure 
is there to enable the current market expansion to continue in an efficient and unfettered manner. 
Infrastructure is characterised by the need for standards in the broadest sense; standards of performance or 
quality as well as of specification. The development of open standards must be encouraged to prevent 
the emergence of de facto standards in the hands of the few. Steps must be taken throughout the 
programme to emphasise the development of the necessary physical and intellectual infrastructure. 
The presence of many users in the programme creates a unique opportunity to do this effectively if the 
projects achieve the right balance between standards, infrastructure, and applications pilots. There is a 
marked trend away from bespoke systems to the use of common platform systems that serve a class of 
applications. Ways must be found in the Programme to encourage the development and use of such 
platforms. The need to encourage work on standards, on infrastructure, and on platforms that help 
to multiply the impact of the individual projects is so strong that consideration should be given to 
making it a necessary condition of every project application that it has at least some element of these 
properties in it. 
3.3 New Systems for the End-user. Companies round the world are working on new, simple devices that 
are aimed at making it easy to access multimedia applications. With such an "information appliance" 
system it will become possible to bring the "unwired masses" into the Information age. New information 
appliances will open the way for new services and leisure activities. The successful information appliance 
system will come in many flavours, but whatever form it takes, software will be the key component. Ways 
must be found in the new Programme to encourage the development and usage of such appliances. 
3.4 Focusing and Integration of Applications. The Programme has shown a welcome flexibility to test 
and then implement new Sectors for new constituencies of users. This has led to a growth of the coverage 
of the Programme, now with 12 Sectors. The increase in the number of Sectors has brought about a 
proliferation of Sector advisory committees, workplan working parties and calls. So, though it is 
appropriate to continue to organise much of the applications part of the Programme in Sectors with 
their own constituencies, the time has come to concentrate the limited resources on: 1) applications in 
those Sectors where the relative immaturity of current use requires attention; and in those few other 
Sectors where there is particularly strong economic or social benefit. 2) the integration of 
11 
1. 
applications arising from many constituencies into common systems with many uses and diverse 
users. This is likely to require special calls, like the one planned for "Integrated Applications" announced 
for December 1996. And above all: 3) the development of standards, infrastructure, and platforms to 
encourage accelerated take-up by the emergence of common approaches. 
3.5 Vision and Strategic Thinking. At all levels of the Programme there is a need for constantly 
adapting vision and strategic thinking to ensure that the fast changing external scene is properly 
considered to direct the limited resources to areas and in ways where they can be of most benefit. 
This will require close collaboration with the policy parts of the Commission. 
4. The ICT Programme as a Whole 
4.1 A Single ICT Programme. For years there has been a degree of friction between the three ICT 
programmes. This has not proved to be a major problem, being handled by good co-operation. In 
particular the co-operation between Telematics and ESPRIT seems to have improved in recent years, but 
the arguments for creating a single ICT programme remain: 1) It is inevitable that there has to be 
considerable interaction between the enabling technology and the systems or application work, and gaps 
must be avoided.. Some parts of the subject still have much fundamental work to be done on them. There 
has to be strong interaction and the boundary will shift all the time. 2) It is undesirable that applicants, and 
especially new participants, should be confused about which programme they should be participating in, 
and even less desirable that they should play off one part of the staff against others. To provide coherence, 
clarity of vision, and drive, a single programme will require a single dynamic, charismatic, and well 
informed leader. 
4.2 A Single Management Structure. A single management structure should be created for the 5th 
FP. The multiple committees of government officials or nominees do serve a valuable role in keeping the 
relevant parts of the national governments well informed. This is particularly important in the area of 
applications. By increasing the coherence between the sub-sections, administrative procedures can be 
improved. The temptation to impose unnecessarily uniform procedures should be resisted. In determining 
the details of the structure one good guide-line is to consider what is the primary constituency that will be 
making applications to work on that part of the programme. So the sub-sections of the Programme 
might be: 
1) Long Range Research. Essentially the old "Basic Research" category but with certain additional fields 
stemming from the needs of applications. This would be the constituency of the advanced, basic ICT 
research workers in the academic and industrial world. The need for long range, research is as strong as 
ever, because of the decline of support in parts of both the public and private sectors in Europe. 
2) Enabling Technology (or Applicable Technology) work. Essentially the ESPRIT and ACTS 
programmes where the constituency is primarily the ICT industries, but without their more user oriented 
work where the users come primarily from constituencies other than the ICT industry. This section can be 
seen as component or sub-systems work. 
3) Applications Systems Development. This covers user oriented work for clear constituencies, other 
than from the ICT industries; the Telematics stream with user oriented streams from the other 
programmes. The work of this sub-section can be seen as system integration and proving work. 
5. Structure of the Applications Systems Section of the Programme 
5.1 The Need for Sub-Structure. The sub-structure for the Applications Systems section should be 
based on natural constituencies of users. Cross sector and integrated applications will need to be 
encouraged. It is not desirable to divide public from private applications in conditions, where public 
bodies can become private bodies; the difference is not significantly larger than that between certain 
constituencies of private sector users. 
12 
1. 
5.2 Reducing the Number of Sectors. Too many sectors creates confusion for the inexperienced 
applicant. The project that spans the boundaries of Sectors is less likely to find favour with the selection 
panels than one that falls squarely in the centre of one Sector's workplan. Yet there are reasons to expect 
that a project spanning more than one application field may be particularly deserving of support. So there 
is every incentive to reduce the number of Sectors. But, on balance, it is better to retain sectors for users 
with a major common interest, than to break down the boundaries and create a continuum with no 
distinguishing features. However there should be calls designed to transcend the Sector boundaries. The 
Panel recommends that existing Sectors should be merged wherever this makes sense in terms of the 
convergence of the constituencies, but that this process should not be pushed too far. The primary 
consideration must lie in the way the constituencies see themselves. A tightly drawn workplan is the 
best way to reduce the over-subscription problem, provided the body that draws up a workplan is well 
informed about the priorities for that constituency. This mechanism should also handle the strategic market 
survey to detect constituencies that deserve a new part in the programme. 
5.3 Cross Sector Co-operation. Steps, such as special calls, have to be addressed to encourage cross-
sector applications and merged or integrated applications. So the Applications sub-programme can be seen 
as consisting of a number of Sectors at one time or call, but as a continuum without boundaries at another 
time or call. The majority of the available funds must still be devoted to the normal Sector 
applications systems projects, but say 10 - 20%, should be reserved for cross Sector calls. 
6. The Challenges 
The challenges for the 5th FP Applications Systems Section of the ICT Programme are a mixture of those 
that have always challenged the programme managers (such as priority and criteria setting, the evaluation 
process in a multi-national environment, project monitoring in a constructive and creative way, and 
fostering exploitation while maximising actions like standardisation work); and those new challenges that 
now face the Programme manager. Amongst these are: 
6.1 User Involvement. The involvement of users is a distinctive and successful feature of the Telematics 
Applications Programme. However, the involvement of users in projects is not without its problems: 
1) Users are frequently inexperienced in participation in R&D projects, unaware of the conventions of the 
"Peer Review" process. Some are more used to political methods rather than the scientific approach. This 
can lead to problems when political pressure is applied to influence application selection. 
2) There is also the danger that the users will be encouraged to become research workers themselves, 
which can lead to poor project performance. 
3) If the users get too much influence over the workplan and project selection process they can be resistant 
to the desirable adoption of innovative new technologies and forward-looking approaches. 
4) Most pilot users are large organisations who may wish to tailor the application too tightly to their 
specific needs, leading to strong risks of over-specification with resulting limited transferability. 
Some ways of minimising these problems are suggested in the main report. 
6.2 Administrative Burden 
1) In the Process. It is inevitable that programmes deploying public funds will be hedged about with 
regulations designed to safeguard the public purse. But these have steadily grown over recent years to the 
point where the programme is being strangled. Many private sector companies are no longer prepared to 
participate. The costs of preparing proposals, the delays inherent in project approval and contract 
finalisation, combined with the relatively small proportion of applications that can be funded, make it 
preferable to forego the advantages in participation. Delegation must be improved. 
13 
1. 
2) In the Project. Once a proposal has been approved and a contract signed, enabling the work to 
proceed, the research worker finds an ever increasing burden associated with the project reporting and 
concertation process, etc. The Commission senior staff claim that some aspects of the burden has been 
reduced in the 4th FP, but the workers seem to perceive an overall increase, perhaps due to the spread of 
horizontal projects. These distractions represent a severe overhead, involving, as they inevitably do, the 
time of senior project staff. Measures must be taken to reduce such an overhead. 
The small high technology firm or the applicant new to the programme are likely to find the overheads and 
delays are unacceptable. Small firms cannot afford to tie up their scarce resources of high quality 
manpower in the overhead activities required of a participant in the Framework Programme projects as 
they are currently organised. It is irresponsible to encourage such a firm to participate. Some suggestions 
for tackling this major challenge of reducing bureaucracy in the Programme are set out in the main report. 
It is not, of course, a problem restricted to the Telematics Applications Programme. 
7. Relations with the Policy DGs 
It is highly desirable that the relevant policy DGs should take an active interest in and be closely associated 
with the relevant work in an Applications Programme. Special mechanisms been established for certain of 
the Sectors. These measures are much to be welcomed, and such mechanisms should be provided to all 
Sectors where policy DGs with an interest in the work are to be found. 
14 
II. COMMISSION'S RESPONSE 
The Commission welcomes this independent evaluation of RTD activities on applications of information 
and communication technologies in areas of general interest. The recommendations will be taken into 
account in orienting the further development of technologies in the Fourth and Fifth Framework 
Programmes and in related standardisation and deployment initiatives. 
Since telematics RTD activities were started at the European Union level in the Third Framework 
Programme, information and communication technologies have developed considerably beyond 
recognition (e.g. Internet, CD-ROMs) and their growing convergence implies that developments will 
steadily continue. Hence, the transition from the fourth to the Fifth Framework Programme presents the 
opportunity to reconsider the Applications component of any future programme concerning information 
society related research, in the light of these technological, socio-economic and regulatory developments. 
The report of the five-year assessment panel offers very useful paths to explore in order to focus the future 
activities on those areas which have the strongest European added value and where there is a genuine 
economic and social case. 
As the report has clearly noted, during the 3rd Framework Programme the RTD work was devoted 
essentially to the development of solutions for the modernisation of services of general interest in Europe. 
The 4th Framework Programme has brought further major objectives, notably to prepare the ground for 
deployment through public-private and public-public partnerships, and to encourage the opening up of new 
markets and the emergence of new companies, in particular innovative SMEs. 
The report correctly notes that the transition from the 3rd to the 4th Framework Programme included three 
major shifts: from the linear model to the feedback model for research, technological development and 
demonstration; from data telematics to multi-media telematics: from technology-push to user-led 
orientation. On the latter point, the report stresses the essential contribution made by telematics RTD 
activities in developing discreet, topic specific, User Constituencies. Such shifts might have deserved a 
stronger emphasis in the report as they resulted in dramatic changes in the work content and in programme 
and project management. They should be re-inforced in any subsequent work in the Fifth Framework 
Programme, along with other shifts, e.g. increased concentration of RTD effort, stronger contribution to 
major EU policy objectives, full implementation of the "contextual research" model (interactive vs. 
linear), systematic involvement of users in projects for continuous socio-economic feedback, more 
comprehensive coverage of all major techno-economic stages (analysis of user needs, functional 
specifications, demonstrator building, laboratory verification, demonstration through field trials, 
exploitation plan containing quantifiable economic criteria for future assessment of success). 
The Commission welcomes the Panel's view that separate series of calls for proposals should aim at 
creating integrated systems with joint usage of two or more different applications, with a view to breaking 
down the sector boundaries. Already under the Fourth Framework Programme, the call to be launched in 
December 1996 on integrated applications should allow the Programme to draw useful lessons on the 
interest, relevance and feasibility of such an approach prior to implementing the next Framework 
Programme. These integrated applications are expected to provide user friendly access to various 
applications from existing platforms (PC, network computer, Kiosks, personal digital assistant, etc.), to 
have an appropriate scale demonstration phase on "digital sites" to allow significant conclusions to be' 
drawn in socio-economic terms, and to enable faster move towards the Information Society. The results of 
this call and the continuous assessment of project implementation will provide essential information in 
support of the preparation of a larger scale operation for integrated applications in the Fifth Framework 
Programme. 
The Commission takes note of the Panel's recommendation to focus the work throughout the Programme 
on building up the physical and intellectual infrastructure, including standardisation work involving 
industry, and to create "platform systems" that can be tailored to the individual user requirement whilst 
allowing the replication of applications across different sectors. The use of such specific industry platforms 
will be encouraged in further RTD activities, drawing usefully on the lessons learned from both the on-
15 
1. 
going sectorial projects and the future integrated applications projects. This approach should certainly 
contribute to encouraging cross sector cooperation, making any further programme function as a 
continuum of activities without boundaries instead of as a collection of distinct sectors. 
The Panel recommendation that preparatory accompanying and support measures as well as "horizontal" 
RTD activities within projects like those of the Telematics Engineering sector, deserve a careful study will 
be taken up. In the meantime, the Commission does consider these as important actions for supporting the 
RTD projects and for fostering the standardisation process and the dissemination and innovation process. 
In order to avoid their becoming what the report calls "overheads projects", their optimal weight should of 
course be assessed, including through consultation of consortia involved in RTD projects, and their 
selection be decided against a clearer strategy and transparent indication of the expected benefits. 
The Commission shares the conviction that greater coordination between RTD programmes, in particular 
through the concept of research/industry Task Forces, and greater flexibility in programme management 
will be beneficial. The "bureaucracy" as it is identified both in the programme management process 
(delays inherent in project approval and contract finalisation, large oversubscription, model contract based 
on cost statements, etc.) and in the management of projects (reporting, "concertation", etc.), has indeed 
been increasingly perceived by participants across the specific programmes as a source of concern and 
even sometimes a source of insurmountable difficulties for the management of Consortia. The first 
orientations for the Fifth Framework Programme, proposed by the Commission under the title "Inventing 
Tomorrow", introduces a simpler structure and more flexible management ideas consistent with the 
recommendations of the 5-year assessment. 
16 
III. EVALUATION PANEL 
Mr. Knud CHRISTENSEN 
Director, R&D and Production, Scandinavian Mobility International A/S. Vice-Chairman, Centre for 
R&D within Rehabilitation (1988-1990). Management consultant with expertise in strategic planning, 
productivity and logistic structures (1988-1990). 
Dr. Max HERRY 
Owner and Managing Director of Traffic Planning Office Herry, Vienna, and of INVOPLAN, Munich. 20 
years experience of national and international transport and telematics projects. Evaluator of 4th FP 
Transport programme (DG VII). President of DANUBE (Exchanges between universities and industry in 
the EU). Project leader or consortium member of 7 finished EU projects, 4 current EU projects. 
Prof. Hannu JAAKKOLA 
Professor for information technology at Tampere University, Finland. Director of research institute for 
information technology (1986-1992). Author of some 80 publications on information technology and 
technology management. 
Prof. Dr. Gunter KROËS 
Professor of national economics, Faculty of Urban & Regional Planning, University of Dortmund. Head 
of Department for National Economies. Expert on regional development in rural areas. Chairman of the 
Telematics Applications programme's external monitoring panel in 1995. 
Dr. Philippe LAREDO 
Associate Research Director, Centre de Sociologie de l'Innovation (CSI) at the Ecole des Mines de Paris 
(since 1987). Responsible for research in programmes in transport economics, Ministry of Transport, 
France (1979-1982). Responsible for national programme on marine technologies and then head of 
evaluation unit, Ministry of Research and Technologies (1983-1989). Evaluation of many EU and Eureka 
programmes. Expert with numerous papers on R&D programmes evaluation. 
Dr. Louis LEE 
Director of information resource services, University of Westminster (since 1989), with responsibility for 
strategic planning, technology and development of academic computing, audio visual resources, libraries 
and management information services to the University. Non-executive Director of an SME that supplies 
library systems and services. Expert on the use of telematics in the university management environment. 
Dr. Jim MOUNTJOY 
Managing Director, Euristix Ltd. Member of the Irish National Information Society Steering Committee. 
Former member ESPRIT Management Committee. Member, Telecommunications Senior Officials 
Committe and of European Space Agency. 
Dr. Brian OAKLEY (Co-chairman and secretary of the panel) 
Retired. Chairman of Logica, Cambridge, (1987-1993). Director of the Alvey programme (1983-1987). 
Secretary of Science & Engineering Research Council (1979-1983). Member of ESPRIT Management 
Committee (1983-1987). Member of ESPRIT Advisory Committee (1987-1992). Chairman of Telematics 
programme mid-term review (1993). 
Prof. Torleif OLHEDE 
Director of the Department of Health Informatics at Spri, the Swedish Institute for Health Services 
Development. Research Associate at the Department of Computer and System Sciences at Stockholm 
University and the Royal Institute of Technology. Expert in security informatics. Swedish representative 
on Health Care Sector Working Party. Head of group responsible for armed forces strategy for IT, 
communications and computer security (1984-1989). 
17 
1. 
Dr. Hans-Detlef SCHULZ 
Managing Director, Media Transfer GmbH, Darmstadt, Germany. Member of Telematics programme 
mid-term review panel (1993). Panel rapporteur, 4th FP call for proposals. Expertise in the field of 
telecommunications and telecommunications standards. Rapporteur for multi-media standards in ITU-T, 
member of ETSI, ISO, and DA VIC. 
Prof. Vincenzo TAGLIASCO 
Full Professor in medical informatics at the University of Genova, Italy. Served as expert for the 
Commission on studies of impact of EU R&D programmes, mid-term evaluation of Human Capital and 
Mobility programme. Expert and writer on the future socio-economic impact of demographic change and 
technology development. 
Dr. Georges WANET 
Member of the Executive Committe of Belgacom since 1992. Head of Corporate Strategy and 
Development. Was responsible for strategic planning, data processing, and the FAST project to improve 
efficiency of basic telephony services. Also Director of F.N.A., grouping 12 telecommunication operators 
to provide managed bandwith services to the Financial Community. Member of Telematics programme 
mid-term review panel (1993). 
Mr. Nils WILHJELM (Co-chairman of the Panel) 
Chief Executive of Industriens Realkreditfond (since 1990). Minister for Industry and Trade, Danish 
Government (1986-1989). Chairman of Federation of Danish Industries (1983-1986). Deputy Chairman, 
Board of Directors of Tele Danmark A/S. 
1996 FIVE-YEAR ASSESSMENT 
ADVANCED COMMUNICATION TECHNOLOGIES 
AND SERVICES 
19 

T. EXECUTIVE SUMMARY 
Europe has excellent 
first-generation 
digital 
communications, 
and a European 
information 
infrastructure is now 
urgently needed. 
Digital video services 
should be introduced 
as quickly as 
possible.. With the Euro-ISDN and GSM standards, Europe has an excellent set 
of first-generation digital communications infrastructures. In terms of 
the technologies used, they are as good as any in the world. However, 
it is vital for Europe to continue to develop communication 
technologies, standardise for interoperation and introduce a European 
Information Infrastructure combining telecommunications, data 
networking and broadcasting capabilities as quickly as possible. 
In particular, digital video services should be introduced as quickly as 
possible. The technology for digital video and television now exists, 
and a programme to accelerate the penetration of digital services 
would liberate valuable spectrum for other services. 
Specific features of a European Information Infrastructure need also to 
be addressed at European level, taking account of the increasing 
importance of the developing Eastern European markets arid the 
Mediterranean Countries. 
with a coherent plan 
and timetable for the 
introduction of a 
European 
Information 
Infrastructure. 
The key objectives of 
the European R&D 
have been met, but 
coherent deployment 
of multimedia 
services is slow. 
Further research 
and technology 
development is 
necessary The cost then of providing broadband services to every household in 
the EU alone will be enormous. Some progress has been made in the 
installation of the core network, but a lot needs still to be done in 
extending this to all citizens and small businesses. Europe badly 
needs a coherent plan and timetable for the introduction of a 
European Information Infrastructure. 
The key objectives of the European R&D telecommunications 
programmes (RACE I / II) have been met. The programme has met its 
technological objectives, but the task of technology development is far 
from being completed. The ACTS Programme has created an 
awareness of the vast potential of digital technology. However, the 
deployment of advanced multimedia telecommunication services on the 
European market has not reached the level which could have been 
expected. The EU RTD has helped interconnectivity and 
interoperability, but legislation may be necessary in some cases to 
ensure coherent deployment. It is also necessary to improve access to 
venture-capital support for innovative high-tech enterprises and start­
ups. 
Further research and technology development is necessary: It will not 
be possible to realise an Information Society in Europe only with the 
technologies of today, and technology development must occupy a key 
position in future EU RTD. Preference should be given to projects that 
lead to improvement of an EII and/or to future concrete commercial 
applications, and should mainly include projects that could have major 
impact on the Union and its population. 
21 
but the focus must 
now shift from 
technical standards 
alone, to trials and 
volume deployment 
EU RTD 
management needs 
improved 
contracting.. 
good programme 
integration... The original rationale for European support to RTD in this area still 
applies, but the focus must now shift from defining technical standards 
alone, to enabling applications and volume deployment on the basis of 
unique and widely accepted standards. European standards are a 
strategic means to maintain the technological initiative in Europe and to 
get a better industrial position in the global marketplace. It is also 
essential that the time span between conceptualisation and the 
realisation of new developments be shortened. Mechanisms must be 
found to achieve real market impact. 
In terms of the management of EU RTD, contracting procedures need 
to be improved and the benefits of using a results-based contract should 
be reviewed, with more emphasis on technical achievement than on 
accountancy. Contracting should be faster and more transparent. 
Independent advice on programme integration and management should 
continue to be sought, and more use should be made of video­
conferencing. 
across programmes. Stronger links should be established with the EU Training and 
Mobility, ESPRIT (information technology) and Telematics Application 
programmes. There is a need to enhance the transparency and co­
operation, including potential joint projects, between these 
programmes. 
and with enhanced 
National initiatives. And the concept of National Hosts needs to be re-evaluated to enhance 
their use. 
In the 5th F. P., 
there should be a 
focus on the 
Communication and 
Information Society, 
with a coherent 
strategy and more 
flexibility. Work in the 5th Framework Programme should be around the theme of 
RTD for the Communication and Information Society. There must be a 
coherent overview of the whole information and communications 
technology development area, with a single coherent EU strategy, a 
single strategic advisory group and flexibility to move resources within 
the whole area as circumstances change. One possibility might be a 
Core Technology Programme and Focused Programmes in specific 
application areas, as described in Section 6.3. 
The main objectives should be to overcome present technological and 
infrastructural constraints to the explicit and hidden needs of users and 
to stimulate trials to demonstrate "in-the-field" the effectiveness for 
users of proposed solutions. It is recommended that use of trial 
infrastructures be enhanced in each Member State. 
Key requirements in 
communications will 
be seamless access, 
ease of access, 
quality of service, 
ubiquitous access 
and service 
customisation.. To enable market development, meeting users requirements, the key 
requirements are for seamless access; ease of access; quality of 
services; ubiquitous access, and service customisation. Particular 
efforts must be made to improve access facilities, improve connection 
facilities, and optimise the interaction between the "access intelligence" 
and the "network intelligence". Standardisation - first for parts and 
then the whole - should remain the primary task: ATM systems must 
also be completely and finally standardised, and multi-media standards 
must be developed jointly by various standardisation groups. 
22 
Enabling Other key areas are in optoelectronics/photonics, microwave 
technologies and propagations, software, interactivity, service engineering, intelligence 
visionary concepts in networks, intelligent agents, mobile communications and in 
must both be information security. 
covered. 
But visionary concepts must also be addressed, some examples of 
which are given in Section 6.5. 
Recommendations 
1. Europe needs a coherent European Information Infrastructure covering all new communication 
requirements emerging from telecommunications, data networking and broadcasting; it is the 
responsibility of the Commission to take the necessary initiatives. 
2. To reach this goal, it is necessary that the process of liberalisation in the telecommunications and 
broadcasting infrastructures and services is speeded up. 
3. As a counterpart, it is also necessary to strengthen and accelerate standardisation. Only unique 
European standards will protect the investments of industry and consumers. 
4. Ways must be found to accelerate the volume deployment of digital TV and interactive multi-media 
services. In addition, opportunities must be sought for exploitation of global niche markets for high-
value business services. 
5. Access to financing for innovation must be improved. The Commission should take initiatives to 
create a better venture-capital financing environment; some options are indicated in Section 1 of the 
Report. 
6. A flexible and functionally effective approach to EU RTD is needed from all European Institutions. 
The Commission must ensure co-ordination, co-operation and transparency in their management of 
programmes. In advanced communications, good programme integration should be continued, and 
with more co-operation with other programmes. 
7. We must prepare now for the 5th Framework Programme, with appropriate and flexible provisions 
for Information and Communications Technology RTD; some of the key issues are highlighted in 
this Report and exploratory work in these areas should already be started. 
8. It is vital to include visionary concepts, as a means of enhancing value chains, and as the seeds of 
future European competitiveness. 
23 
II, COMMISSION'S RESPONSE 
The Commission welcomes this independent evaluation of EU RTD in the broader perspective of advanced 
communications developments in Europe. The recommendations will be taken into account in orienting the 
further development of technologies in the 4th and 5th Framework Programmes and in related 
standardisation and investment stimulation initiatives. 
The Commission accepts its responsibilities for catalysing the development of a coherent European 
Information Infrastructure covering all new communications requirements and addressing the specific 
features of European regions, Eastern European and Mediterranean countries markets. The technology 
developments in the RACE (1988-1994) and ACTS (1994-1998) programmes are directed to support inter­
operability in the convergence of telecommunications, data networking and broadcasting areas. In 
addition, coherence and interoperability are key features in the new regulatory framework for 
communications in the Information Society. 
Liberalisation and competition in provision of infrastructure and services has been a key objective of EU 
telecommunications policy in recent years. The Commission will seek to ensure that the process of 
liberalisation in telecommunications is followed through by Member States on the agreed timetable. 
The multiplicity of infrastructure and service providers in a competitive market, and the new diversity of 
technologies for service provision, will increase the need for effective standardisation and consensus 
development. The EU RTD programmes, both RACE and ACTS, have made major contributions to 
technical standardisation - as noted in the 5-year assessment. Further support will be provided from the 
remainder of the ACTS programme in the form of guidelines and contributions to industry-led 
standardisation. The need for substantial EU support to standardisation will be considered in preparing the 
Commission's proposal for Information Society RTD in the 5th Framework Programme. 
The volume deployment of digital TV and interactive multi-media services is primarily a responsibility of 
the private sector. Nevertheless, the new framework for EU investment stimulation in Transeuropean 
Networks will be used by the Commission to catalyse the coherent and simultaneous roll-out of new 
services within the EU. Exploitation of global niche markets for high-value business services will be 
facilitated by emerging "electronic commerce" systems. The Commission will implement a new Co­
ordinated Electronic Commerce Initiative in 1997, building on the RTD in the ACTS programme and the 
Memorandum of Understanding recently signed by leading industrial interests and SME associations. 
In 1996, the Commission published a Green Paper on Innovation, and has adopted an Action Plan on the 
basis of the reactions from Industry and Member States. The Green Paper covered the improvement of 
innovation financing, particularly for new technology-based firms. The options in the Strategic Audit 
Report will be given careful consideration. 
The Commission shares the conviction that greater co-ordination between RTD programmes, and greater 
flexibility in programme management will be beneficial. The first orientations for the 5th Framework 
Programme, proposed by the Commission as "Inventing tomorrow" , introduce a simpler structure and 
more flexible management ideas consistent with the recommendations in this "Strategic Audit". A re­
inforced provision for visionary research at European level will allow many of the new ideas in this report 
to be taken forward. 
COM(96)332 Final of 10.07.96. 
24 
Ill, EVALUATION PANEL 
Mr. Stelios Argyros MEP 
As a Member of the European Parliament, Stelios Argyros sits on the Committee on Research, 
Technological Development and Energy and the Committee on Economic and Monetary Affairs and 
Industrial Policy. He has served on the Board of numerous companies in Greece, has been Chairman of 
the Board and President of the Federation of Greek Industries and is currently a Vice President of UNICE. 
Mr. Max Artigalas 
General Manager Recording Programme Administration of THOMSON multimedia. After being technical 
manager of Thomson Broadcast, he joined THOMSON multimedia where he is also recording domain 
manager for Corporate Research and Advanced Development. 
Dr. Hans Baur 
Former Executive Vice President of Siemens AG, and now acting as an advisor to several national and 
international bodies and to Mr. Bangemann on the Information Society. During his career with Siemens 
AG, Dr. Baur was a member of the Corporate Executive Office and was responsible for the business 
groups dealing with telecommunications and electronic components. He also served as President of the 
Supervisory Board of Siemens Nixdorf Informationssystems AG. 
Mr. Umberto de Julio 
Co-General Manager of STET, with responsibility for strategic planning, technology and development of 
new service, prior to joining STET in 1995, Mr. de Julio held the position of Director of the Network of 
Telecom Italia in 1994. His earlier career with SIP began in 1972, where, in 1986, he was appointed 
Director of the Network. He is a Member of the Board of Telecom Italia, Telecom Italia Mobile, CSELT, 
STREAM, STET International and is also President of the General Assembly of EURESCOM and Vice-
General President of AEI. Mr. de Julio was assisted by Mr. Giovanni Perucca, Director of R&D of 
STET. 
Mr. Aimo Eloholma 
Vice President of Telecom Finland Ltd., Sales and Marketing, since 1995, Mr. Eloholma was formerly 
Vice President of Telecom Finland, Business Services, and has held a variety of positions within the PTT 
in Finland. He is Chairman of the Board of Tele Yritysviestintà Oy (provider of PBX's) and Yritysverkot 
Oy (Corporate Networks Ltd.) and serves on a number of Committees in Finland. He is also a member of 
Information Society Forum (European Commission). Mr. Eloholma was assisted by Mr. Timo Rajamaki, 
Director of the Research Centre of Telecom Finland. 
Dr. Francisco Pinto Balsemâo 
Chairman and majority share holder of IMPRESA, S.G.P.S., a holding company with major positions in 
newspapers, printing, advertising and a Portuguese private television channel. He is Professor of 
Communication Science at the New University of Lisbon, Chairman of the European Institute for the 
Media, Vice-President of "Fondation Journalistes en Europe" and Member of the Board of the Media 
Business School. He was one of the founders (1974) of the Popular Democratic Party (PPD), now the 
Social Democratic Party (PSD) and from 1981 to 1983 he was Prime Minister of Portugal. Dr. Pinto 
Balsemâo was assisted by Mr. Antonio Trigo de Sousa, Technical Director in SIC. 
Dr. George T. Waters 
Director of the Technical Department of the European Broadcasting Union since 1986, Dr. Waters has 
been involved in broadcasting for more than 35 years. He completed a seven-year term as Director 
General of RTE, Ireland's national broadcasting organisation, and was a Director of the International 
Council of the Academy of Television Arts and Sciences (USA) and Vice President of the European 
Broadcasting Union. In 1993 he was appointed President of the International Academy of Broadcasting 
(IAB). 
25 

1996 FIVE-YEAR ASSESSMENT 
INFORMATION TECHNOLOGIES 
27 

I. EXECUTIVE SUMMARY 
1.0 Scope of the Assessment: Lessons for the Fifth Framework Programme 
The Panel's five-year assessment of the European Strategic Programme for Research in Information 
Technologies (ESPRIT) is based on an examination of the many documentary materials supplied by the 
Commission, including the 1996 report of the ESPRIT Review Board produced under the chairmanship of 
Professor Umberto Colombo. It has also interviewed senior Commission staff concerned with Information 
and Communications Technology (ICT) policies. In addition, the Panel sought the opinions of national 
delegates to ESPRIT. 
The review has focused on activities carried out under the European Community's Third Framework 
Programme, which ran from 1991 to 1994, and during the initial period of the Fourth Framework 
Programme, up to early 1996. This assessment incorporates the final evaluation under the Third 
Framework Programme. 
The Panel has especially focused on some specific features of the present ESPRIT programme and, in 
agreement with the Commission, on strategic issues affecting ICT research and technology development in 
the Fifth Framework Programme. It has arrived at a significant number of constructive recommendations 
that reflect the Panel's view of the crucial role of ICT and of the importance of the Information Society. It 
hopes that the Commission will endorse this judgement and assign a budget to the future ICT programme 
in the Fifth Framework Programme which is at least equivalent to the present one; that is, about one third 
of the overall Framework Programme Budget. 
2.0 A Brief History of ESPRIT: Steadily Widening Objectives 
Community funding of IT R&D begun in 1984 . During the first decade, the programme has been 
essentially aimed at closing the gap between Europe's information technologies industries and those of the 
US and Japan. 
The strategy changed radically with the launching of the Fourth Framework Programme in 1994 which 
recognised the increasingly critical role played by IT in the competitiveness of all industry. The new IT 
Programme: 
promotes vertical collaboration between ICT users and suppliers; 
gives much closer attention to focusing results on market needs; 
facilitates access to new technologies and their take-up; 
encourages an awareness of emerging R&D results; 
employs a number of new tools to implement the programme such as networks of excellence and 
focused clusters (activities covering different technological areas with a single well-defined goal). 
Council decision 84/30 (28.02.84) established ESPRIT as part of the First Framework Programme (1984 - 1988) 
with a budget of ECU 1,500 million. Council decision 88/279 (11.04.88) continued ESPRIT into the Second 
Framework Programme with a budget of ECU 3,200 million and Council decision 91/2328 (15.07.91) renewed 
ESPRIT as part of the Third Framework Programme with a budget of ECU 2.035 million. Council decision 
94/802 (23/11/94) established a new programme, called Information Technology Programme, as part of the Fourth 
Framework Programme with a budget of ECU 1,911 million. 
29 
3. 
3.0 The Context: Europe's Transition towards the Information Society 
The Panel's report is written against the background of rapid change in the importance, development and 
use of ICT. The moment is opportune to present some new ideas enabling Europe to adapt to at least four 
key developments: 
• Europe's transition to the Information Society has begun and is a crucial challenge for Europe's future 
economy and society; 
• deregulation of European telecommunications, which should be a major factor aiding this transition, 
is now little more than a year away; 
• rapid growth in use of the Internet is bringing new opportunities to businesses and citizens; 
• multimedia technologies are penetrating the worlds of business, the media, public services and 
education. 
This report urges radical changes in the organisation and design of ICT programmes to enable the EU to 
respond better to these developments. 
4.0 Priority Issues: A Single ICT Programme Which Brings More Products to Market 
Two priorities have been identified and addressed in the Panel's recommendations: 
4.1 The need to merge the ACTS, Telematics and ESPRIT programmes 
The Panel has come to the conclusion that the objectives of these separate programmes would be better 
achieved if they were brought together in a single programme that covers all aspects of ICT required by 
the Information Society. 
Given the fact that ESPRIT operates on the basis of annual work programmes, the Panel believes that 
many of the changes in this direction can be implemented on a step by step basis during the remainder of 
the Fourth Framework Programme. This will help to smooth the transition to the Fifth which begins in 
1999. 
An attempt to define a timetable for the programmes' merger would have been outside our terms of 
reference. But we believe it would be helpful to establish a joint coordination committee between the three 
programmes to prepare for the event. We are aware that we are advocating a profound change which 
needs careful planning and, above all, avoidance of administrative obstacles to what would be a significant 
improvement in Europe's programmes for ICT. 
4.2 The need to achieve much swifter commercial applications of research results 
The panel has concluded that ESPRIT's programme objectives have been satisfactorily accomplished. 
Among other things, it has made a vital contribution to promoting cross-border collaboration between 
small and large industries, universities and research institutes. 
However, the programme could be even more effective if - in addition to activities such as best practice, 
first-user actions and assessments - better mechanisms were established for ensuring the commercial 
exploitation of results. A number of actions could be envisaged, including: 
• focusing results much more explicitly on the tasks needed to construct Europe's Information Society; 
• initiatives to promote the commercial exploitation of research results; 
30 
3. 
• developing the tools to measure and monitor European ICT development, take-up and market shares; 
• encouraging SMEs with innovative capacities to strengthen and speed up their marketing of new 
value-added products; 
• improving the efficiency and accountability of the programme's administration, partly by making it 
more results-oriented; 
• define a long-term strategy for European research and technological development in ICT. 
5.0 The Panel's Main Findings 
The Panel's findings look back to the five years of the assessment period and forward to the needs of the 
future programme. 
5.1 Assessment Period 1991-1996 
5.1.1 Programme Objectives: Time to Add The Needs of Society At Large 
The Panel agreed that ESPRIT's objectives have been relevant and fully endorsed its new focus on 
responding to the needs of business and industry as a means of boosting competitiveness and employment. 
However, there is now an additional need to broaden the objectives to include the needs of society at large 
which must find their response in the evolving infrastructures, services and applications of the 
Information Society. 
Having assessed the programme's performance against its objectives, we would draw particular attention 
to: 
• the positive change in the dynamics and perspective of ESPRIT from the Third to the Fourth 
Framework Programmes; 
• the focus on relevant topics has been made possible by a broad description of research tasks and a 
continuously updated work programme; 
• an imbalance between the user-need, market-driven priorities and the requirement for long-term 
research. The new focus is welcome but we are concerned that the necessary balance with long term-
research is being jeopardised. 
5.1.2 Programme Efficiency: Relevant Effort, Still Some Shortcomings 
The Panel considers that the management as a whole has made great efforts to raise the programme's 
performance and efficiency. In addition, worthwhile changes have been introduced to make the 
Programme more flexible - a feature which must be characteristic of the Fifth Framework Programme. 
However, the search for flexibility has turned on attempts to satisfy the specific needs of proposers and 
this, paradoxically, has made certain aspects of the programme very complex (e.g. variety of actions and 
information packages, different types of proposal submissions). 
The Panel also wishes to draw attention to the more general problem of administrative delays which 
characterise the Framework Programme as a whole, particularly in concluding contract negotiations and 
making payments. 
5.2 The Future of ESPRIT: Five Challenges for the Fifth Framework Programme 
The Panel has identified five areas that must be addressed in the Fifth Framework Programme: 
31 
3. 
(i) Programme Strategy and Design 
(ii) Programme management and evaluation 
(iii) The Information Society 
(iv) Competitiveness and exploitation of research 
(v) SMEs and start-up companies 
5.2.1 Programme Strategy and Design: Three Into One Will Go 
Currently, there are many overlaps between ESPRIT, ACTS and Telematics which need to be turned into 
synergies. This could best be achieved by merging these three ICT programmes into one. Within this 
single programme a proper balance must be struck between development of close-to-market innovative 
applications and long-term research. 
The Panel's detailed recommendations envisage groupings of applications and technologies within one ICT 
programme. This would have two clear purposes: to stimulate innovative applications of emerging 
technologies and to satisfy the requirement for strategic and long-term research. 
5.2.2 Programme Management and Evaluation: In Need of Overhaul 
Every assessment of the Framework Programme and of its constituent elements as well as a survey of 
national opinion undertaken by the Panel confirms that the programme management has become more 
efficient. Nevertheless, the Panel supports the widespread view that there is still considerable room for 
improvement. 
Oversubscription is a familiar problem with Community-supported initiatives and it imposes heavy costs 
and burdens on the proposers. It is important to resolve the difficulty because it causes frustration and 
dissatisfaction and could prompt dynamic companies to abandon their interest. The limited introduction in 
ESPRIT of the two-step procedure has proved an extremely useful attempt to deal with the situation . The 
Panel recommends extending it to the whole programme. 
The Panel's detailed recommendations seek to make the future ICT programme an efficient instrument for 
Europe's ICT community in which delays would be cut by a half, and project performances much more 
strictly evaluated. 
5.2.3 The Information Society: ICT as a Tool for Economic Growth, Social Cohesion and 
Personal Development 
As the Information Society evolves, key aspects of political, economic and social life will be increasingly 
permeated by the use of ICT and its applications. If Europe handles the transition in a timely fashion, it 
will significantly strengthen its competitiveness in global markets and cash in on faster economic growth, 
more employment and a better quality of life. 
By contrast, if it continues to move more slowly than some of its competitors, then our countries will find 
it very difficult to maintain, let alone increase competitiveness across the broad range of their economies. 
A very serious obstacle to the realisation of the Information Society is that European businesses and 
individual citizens are lagging far behind the US and some other countries in embracing ICT and in 
exploiting the opportunities it offers for both corporate and personal growth. The need to integrate the 
use of ICT into education and training is paramount, both to establish computer literacy and as a tool for 
learning. 
32 
3. 
The Panel's detailed recommendations seek to ensure that the future ICT programme is a key pillar in the 
construction of the Information Society. It should support projects which help to deliver public goods 
serving citizens' needs, which act as a catalyst for user-friendly applications and which foster a growing 
market for the new technologies. 
5.2.4 Competitiveness and Exploitation of Research: The Former Grows if the Latter 
Succeeds 
Innovative development and innovative use of ICT has to be both visionary and focused if Europe is to 
maintain and enhance its competitiveness in the global market place. Competitiveness creates employment 
and both can be secured if Europe nurtures the development and take-up of a world-class ICT that both 
produces and uses ICT to offer constantly improved applications and services. 
Constant improvement would result from a better commercial exploitation of the results of ESPRIT 
projects. In cases where an attempt is made, windows of opportunity are frequently missed because the 
process has been far too slow to adjust to rapid changes in the market place. The Panel believes that the 
existing eligibility criteria, combined with various practices, has sometimes led to the creation of 
somewhat artificial teams lacking clear commercial objectives. Relaxing some of the criteria and insisting 
on common commercial objectives would certainly improve the situation. 
The Panel's detailed recommendations aim at more benchmarking of ICT-based competitiveness in the 
future programme, measures to ensure a shorter time-to-market, encouragement of projects which have 
an entrepreneurial final objective and supporting measures to help commercial exploitation of results. 
5.2.5 SMEs and Start-up Companies: Finding the Right Responses to Particular Needs. 
Small and medium-sized enterprises account for around 80% of Europe's employment. ESPRIT is well 
able to provide specific support to SMEs and start-up companies, but its full potential is not being 
exploited. 
Even though the SMEs represent 30% of the participants, the programme would benefit from more 
simplicity to encourage SMEs that have made serious efforts to develop commercial ICT applications. Too 
few awards are made to SMEs to enable the sector to achieve the commercial opportunities open to it. 
The Panel's detailed recommendations seek to endow the future ICT programme with procedures which 
are more user-friendly for SMEs, structures which are more supportive, funding rules which are more 
flexible and special measures for encouraging risk taking and for helping start-up companies. 
Recommendations 
All of the following recommendations have been designed to apply to the future single ICT programme, 
except for group A which concern the Framework Programme as a whole. 
A. Improving the Framework 
A.l Cutting Delays 
a) The Commission should overhaul its current administrative functions and re-engineer key 
procedures in order to cut delays by half. Payment procedures in particular must be 
streamlined to reduce the time from submission of an invoice to actual payment from the 
current 90 days to 30 days. 
33 
3. 
b) The key administrative procedures should be reviewed to speed up decision making by 
delegation of authority to appropriate levels and by reduction in the layers of decisions. 
c) The project partners should be allowed to charge for expenditures from the day of the 
project's approval so as to cover their costs during the contract negotiation phase. A down 
payment should be made immediately upon contract signature. 
d) The current payment scheme based on statement of costs should be abandoned in favour of a 
deliverables-based payment scheme. This will not only simplify procedures but also 
encourage more efficient and productive projects. 
e) In order to raise efficiency, timeliness and effectiveness, some of the non-strategic 
management tasks undertaken by Commission staff should be wholly or partially contracted 
out to competent external organisations. 
A.2 Effective post-project evaluation. 
A simple method must be developed to evaluate projects after their completion, and a user's guide must 
describe how and when to apply this method to different types of projects. Contractors must be obliged to 
provide the Commission with the necessary information to permit effective post-project evaluation. 
B. Programme Strategy and Design 
B.l Uniting Strengths. 
a) Separate programmes for Telematics, ACTS and ESPRIT should be abandoned and be 
replaced with groupings of applications and technologies under one integrated ICT 
programme. This is what is necessary to bring together all the technologies required to build 
the Information Society. 
b) In allocating funds in the Fifth Framework Programme, the Panel strongly recommends to 
assign a budget in line with the total ICT programmes in FPIV and not below one third of the 
overall FPV budget. 
B.2 Facilitating New Ideas. 
a) The new ICT programme must emphasise the need for Europe to become more innovative by 
stimulating the development of innovative applications using emerging and enabling 
technologies. 
b) Exchanges of researchers, corporate engineers and professionals from hospitals, schools, 
public services, etc., should be encouraged to create a better understanding between 
technology inventors, developers, and users. This would improve the rapid application of new 
ICT developments. 
B.3 Incorporating Vision. 
a) A strategic vision of the future organisation and functioning of Europe's Information Society 
must be constantly developed and reviewed. 
b) An "ICT Foresight Board" should be set up to help develop a common European vision on 
long-term research and market trends. 
34 
3. 
c) Long-term research plays an important part in the shaping of Europe's future and must be 
given a secured place in European research and technology development in ICT. Around 15 
% of the programme budget should be reserved for basic long-term research, strategic 
research, and radical innovation. 
d) As the scope of the future Framework Programme broadens to address the needs of society at 
large, the European Parliament and the Council of Ministers should play a more active role 
in defining strategic goals, evaluation criteria and methods, assessing the results of 
programme evaluations, and steering the course of the programme. 
C. Programme Management and Evaluation 
C.l Serving the Client. 
a) The future ICT programme must be a more efficient instrument to the European community 
of ICT users and suppliers. 
b) To reduce the overwhelming volume of applications, the two-step proposal, introduced in a 
limited number of areas of the programme, should be the norm for all areas. This should be 
based on a short 2 -4 pages pre-proposal which would be followed by a full proposal only if 
the pre-proposal was accepted. 
c) The Focused Clusters, introduced for the first time in ESPRIT under FPIV, should be more 
widely used in the next ICT programme. 
C.2 Increasing Accountability. 
a) The future Framework Programme should set specific goals for European ICT development, 
take-up and market shares, and for a target growth in employment. 
b) Indicative measures of how the programme is succeeding in building the Information Society 
should be established to demonstrate the effectiveness of the programme to its stake-holders: 
the Member States, the regions and their citizens. 
D. The Information Society 
D.l. Enabling the Information Society. 
a) An Information Society Cluster should be organised within the future ICT programme for 
promoting the development and take-up of IS technology in such areas as content creation, 
information management, applications for citizens, communities and companies. The Cluster 
should be allowed to receive ideas and project proposals on a continuous basis. 
b) Proposals should be given a priority that bring together ICT suppliers and users in joint 
efforts to define, develop, deliver and take up IS technology and services. 
c) A single point of contact for Information Society projects should be established at the 
Commission with particular responsibility for coordinating the allocation of ICT programme 
finance with the structural funds. 
35 
3. 
D.2. Networking Citizens and Communities. 
a) Pilot projects should be identified to foster understanding and acceptance of the Information 
Society and to encourage the use of ICT in every day life. Special attention should be given to 
those who may have particular difficulties of access to technology or in using it, such as the 
elderly, the unemployed and the handicapped. 
b) Closer links aimed at promoting the use and understanding of ICT must be established 
between the programme and authorities responsible for education and training in Europe, 
typically at the regional level. 
c) National and regional authorities together with non-governmental and professional 
organisations (e.g. teachers and doctors) should be encouraged to submit project proposals, 
preferably in vertically integrated teams involving all players in the ICT value chain, from 
suppliers to users. 
D. 3 Preparing the Citizen for the Information Society 
a) Strong encouragement should be given to specific projects targeting employment issues, such 
as training, retraining and life-long learning technologies or applications. In addition, the 
likely impact of a proposed project on employment should be included in the evaluation 
criteria. 
b) The future ICT programme should support the establishment of appropriate technical 
standards not only to encourage business and market development but also to protect citizens' 
rights, particularly in the fields of the multimedia content industry and electronic commerce. 
c) The programme shall encourage and support projects that specifically address the issues of 
low-cost access to public information. 
d) The future ICT programme must encourage and support projects aimed at improving the user 
interfaces of ICT applications in general, and of training systems in particular. 
E. Competitiveness and Exploitation of ICT 
E.l. Establishing Milestones 
a) A long-term strategy for achieving and sustaining European competitiveness through the 
development and take-up of ICT must be developed by the Commission in collaboration with 
industry. The strategy must include clear objectives, market-share targets, and the expected 
contribution from RTD. A unit should be set up to monitor how Europe is progressing 
towards its goals. 
E.2 Shortening Time to Global Markets. 
a) The programme must encourage industry to develop and bring ICT applications to the market 
in shorter time. 
b) It must emphasise collaborative efforts in commercial exploitation and facilitate a market-
driven approach by shortening the life of projects so that time-to-market periods are between 
1-2 years. 
36 
3. 
E.3 Selecting the Fittest. 
a) The programme must support those who can and will exploit the results of European funded 
R&D, and discourage those partners who have no real interest or capacity for exploitation. 
b) More efficient models of cooperation should be encouraged, with smaller partnerships built 
around real interests. Every partner in a team must have a distinctive and necessary role. In 
evaluating proposals, the team efficiency and the potential for commercial exploitation of the 
project results must be given higher weighting. 
E.4 Moving Exploitation to the Top of the Agenda. 
a) The Commission should set up a Help Desk including a network of experts in commercial 
applications, marketing, industrial and trade law, banking and venture capital to provide 
coordinated support for commercial exploitation and to encourage benchmarking. 
b) An Innovation Financing Help Desk should be set up and linked to the Framework 
Programme as proposed in the Commission's Action Plan for Innovation. 
c) The Commission should foster the creation of an advisory panel involving major players with 
a decisive influence on the development of an effective single European ICT market (utilities, 
PTTs, regulators etc.). 
d) The Commission should establish and fund Reference Sites where European applications can 
be demonstrated. Such demonstrations must be well publicised and accessible via the WWW. 
Sites to facilitate local access should be established in large potential markets in the US, 
Japan and the Far East. 
e) Priority must be given to achieving a more rapid penetration of world-wide markets by 
helping companies to set up subsidiaries and joint ventures in developed markets. These 
should be treated as start-up companies for the purpose of RTD in the new Framework 
Programme, provided they bring employment to Europe. 
F. SMEs and Start-up Companies 
F.l Making Life Easier. 
a) The Commission should establish an SME Desk as a single point of contact between SMEs 
and the programme. The purpose of the Desk should be to provide full support and to 
eliminate the present difficulties companies have in navigating through the programmes. The 
Desk would evaluate ideas for proposals and provide all of the necessary information on what 
is required of a proposer and when. 
b) Create a new category of Small and Medium-sized Project (SMP) aimed mainly, but not 
exclusively, at SMEs which runs for less than one year and with Commission funding not 
exceeding ECU 500,000 per project. 
c) Fast-track support should be established for Small and Medium-sized Projects, allowing 
adoption of unsolicited proposals not linked to a particular call. The services of the SME 
Desk would be at the disposal of SMEs wishing to pursue a small project. 
d) Some future ICT programme funding - around 5% - should be earmarked for special calls 
aimed at stimulating partnerships in which SMEs take the leading role in the development and 
37 
3. 
exploitation of results. The adequacy of the funding should be judged in the light of the 
scheme's success in creating these highly desirable partnerships. 
e) Efforts should be made to thoroughly familiarise SMEs with the future ICT programme and 
its objectives by involving them more closely in the consultation process and by extensive use 
of external evaluators and project reviewers drawn from SMEs. 
F.2 Matching Market Endorsement. 
a) Programme funding must be made more flexible in order to provide specific incentives to 
SMEs with exploitable ideas. 
b) Funding should be increased to 60% when the potential commercial validity of an exploitable 
project has been endorsed by venture capital backing, or by a large company covering at least 
10% of a project's costs without itself participating. 
F.3 Stimulating Start-Ups. 
a) Maximum funding of 75 % should be available for projects with good exploitation prospects 
involving start-up companies which are not subsidiaries of other companies. 
b) Grants should be available to start-up companies which are partners in successful projects to 
cover the costs of finding and securing investment capital for commercial exploitation. These 
could be 10 - 15 % of the amount that the EU contributed to the start-up's part of the project. 
38 
IL COMMISSION'S RESPONSE 
The independent evaluation of the programme over the last five years was prepared by the high level 
group of experts chaired by Professor Roberto Carneiro. The evaluation was based on the examination of 
documentary material provided by the Commission including the final report produced under the 
chairmanship of Professor Umberto Colombo which evaluated the activities supported by the programme 
under the previous Framework. The Commission welcomes the recommendations of the panel which 
essentially address the future orientations of the programme. 
The Commission views the main findings and recommendations as generally useful for the remaining 
period of the current Framework, and for acting as guidelines for the current discussions associated with 
the preparation of the 5th Framework Programme. The Commission welcomes the recommendation that a 
number of new approaches introduced in the current programme (e.g. rolling workplan, regular calls, 
two-step evaluation) be generalized as appropriate in the Fifth Framework Programme. 
The need for combining various technologies and actions towards the establishment of an "user-friendly 
Information Society" has been stressed in the Communication "Inventing Tomorrow". The Commission 
welcomes the panel's views on this topic and shares its conviction that greater convergence of efforts is 
necessary and that a single and integrated programme addressing Information and Communication 
Technologies is required. As suggested in the executive summary, the programme will take advantage of 
its rolling workprogramme to start implementing changes during the Fourth Framework Programme to 
smooth the transition into the fifth. 
The Commission supports in particular the view that technology take-up should be further emphasised in 
the forthcoming programme to imrove competitiveness and create employment through a more wide­
spread exploitation of ICT. The Commission welcomes the panel's emphasis on the role of citizens and 
communities. Technological progress in the ICT area increasingly depends on dynamic markets for new 
products and services and hence on the ability and willingness of citizens and workers to use new ICT. 
The balance between long-term research and take-up activities needs to be defined according to the 
industrial and socio-economic objectives of the forthcoming programme. A strategic and forward-looking 
vision of research and social developments is certainly required, as mentioned in the "action plan for 
innovation" to identify promising technologies and markets. 
The Commission welcomes the recommendations aiming at improving SMEs' access to the programme. 
The suggestions made by the panel support the "Inventing Tomorrow" proposal to help SMEs access the 
programme by developing technology transfer mechanisms and by establishing links with venture capital 
and financial engineering tools. 
Finally, the Commission supports the view that, at the Framework level, additional efforts need to be 
made to improve the flexibility associated with, and the efficiency of, administrative procedures, to 
shorten the deadlines and administrative costs so that dynamic companies, in particular SMEs, are further 
encouraged to join the future programme. 
39 
TÏT. EVALUATION PANEL 
Prof. Roberto Carneiro (Chairman), President of TVI, Televisào Independent S.A. (Board of 
Shareholders) Group Forum, SGPS. 
Former Minister of Education and Sports in Portugal (1987-91), senior World Bank (IBRD and IDA) 
consultant for projects in Latin America and Africa (1975-87); and since 1978 OECD senior consultant on 
education policy, manpower planning and public management; and UNESC senior consultant and expert 
on education and training policy. Recent international activities include consultant coordinator to SIGMA, 
EEC/OECD joint venture designed to implement State reform in Central and Eastern European Countries 
(1992-93); OECD Examiner of the French Educational Policy (1993); Member of the International 
Commission for Education in the 21st Century (UNESCO, 93/95); vice-president of the Information 
Society Forum and Member of the Bureau which coordinates the European Commission Study Group on 
Education and Training under the auspices of Commissioner E. Cresson. 
Andrew Bos well, Managing Director, Enterprise Technology and ICL Technical Director. 
Chairman of the UK National Information Infrastructure Task Force, which works with the Department of 
Trade and Industry on ICT related issues. Appointed to the G7 Committee responsible for accelerating 
standardisation in ICT, an initiative that stemmed from the G7 summit of 1995. 
Jean-Michel Chassériaux, Managing Director ofERCIM and Director of International Relations 
atlNRIA. 
Scientific attaché to the French Embassy in London (1973-77), senior consultant for the Ministry of 
Foreign affairs and the Ministry of Industry (1977-84). Deputy-director for training, research and regional 
affairs at the ICT industries' Unit of the Ministry for Industry (1984-87). Director for international affairs 
at the Ministry of Research and Space (1988-93). Professor at the Paris 7 University since 1983. 
Bernt Ericson, Vice-President, Research and Technology, Telefonaktiebolaget LM Ericsson. 
Joined Ericsson in 1969 as Chief Engineer and member of the Business Strategy Group; Department 
Manager for special projects within the Strategic Product Planning division; Director of applied Research 
and finally, vice-president. Elected as member of the Royal Swedish Academy of Engineering Sciences 
and of the European Science and Technology Association(ESTA) of the European Commission. 
Roberto Galimberti, Chairman and CEO of Etnoteam. 
From 1968 to 1973 head of Digital Computing Division of Laben. With Italtel as Research and 
Development Director and later as head of the Toll Switching Division. Since 1982 Chairman and CEO of 
Etnoteam. Since 1994 Chairman and CEO ofl.NET, a leader in Internet business connectivity. Also vice-
president of CED Borsa (Italian Stock Exchange Information System Company), and of Intercai Etnoteam. 
Member of the board of Anasin (Italian Computer Services Association). 
Hartmut Raffler, Senior Director, Siemens Corporate Technology Division. 
Joined Siemens AG in 1979. Responsible for the Innovation Project Communications. From 1993 to 1996 
Head of the Software and Engineering Department. Since April 1996 Head of the Information and 
Communications Department. Main topics: Information and Media Technologies, Networks and Video 
Communications, Security in Systems, Knowledge Processing and Man-Machine Cooperation. 
Josep Maria Vila Solanes, General Manager of INDRA Group, President of INDRA SSI and 
President of INDRA SCA. 
Chief Executive Officer of ERITEL, General Manager of Olympic Games Organising Committee at 
Barcelona '92. Deputy Manager of ENHER and Branch Manager of UNISYS. Director of the 
DATAMATION magazine (Spanish edition). Professor in the Nuclear Technology Department, 
Universidad Politécnica de Barcelona, and formerly Professor in other universities and research centres in 
Barcelona. 
40 
3. 
Herbert Wotke, Managing Director of the Austrian Industrial Research Promotion Fund (FFF). 
Served for more than 2 decades in the FFF (Forschungs Fôrderungs Fonds fur die gewerbliche Wirtschaft) 
in national and international funding activities including technology policy, EUREKA and the 
governmental "Innovation and Technology Fund". 
41 

1996 FIVE-YEAR ASSESSMENT 
INDUSTRIAL MATERIALS AND TECHNOLOGIES 
43 

I. EXECUTIVE SUMMARY 
1.0 Evolution of the Industrial and Materials Technology Programmes 
Among European decision-makers, there is agreement that industrial and materials technologies are of key 
importance to the future success of European manufacturing industry. The present IMT programme, 
alternatively known as 'Brite-Euram', stems from two RTD programmes, 'Brite' and 'Euram', conducted 
under the First Framework Programme. Brite was concerned with a wide range of industrial 
manufacturing technologies, while Euram was devoted to advanced materials. In the Second Framework 
Programme(FP2), the two subject areas were brought together in 'Brite-Euram', and work related to 
aeronautics was added. This arrangement has continued, with the further addition of surface transport 
technologies in the current Fourth Framework Programme. 
The central emphasis of the work under these programmes has been on precompetitive, generic research, 
conducted in a collaborative manner across national borders and involving industry, research institutions 
and universities. Throughout, an aim has been to strengthen the scientific and technological basis of 
European industry and encourage it to become more competitive at international level. Particular attention 
has been paid to the support and encouragement of SMEs, and to the development of European economic 
and social cohesion. 
Successive IMT programmes have been expanded in funding and scope. FP4 IMT is considerably larger 
than FP3, and has a broader range of objectives including improving the quality of life while promoting 
sustainable economic growth. IMT funding is summarised below. 
FP1 FP2 FP3 FP4 
220 MECU 500 MECU 748 MECU 1617 MECU 
(Brite + Euram) (Brite-Euram) (Brite-Euram II) (Brite-Euram III) 
2.0 Scope of this Five-Year Assessment 
This Five-Year Assessment covers the Industrial and Materials Programmes carried out under the Third 
Framework Programme( 1991-94) and in the first year of implementation of the Fourth 
Framework Programme( 1994-98) 
In accordance with the Terms of Reference, the assessment evaluates three principal aspects of the 
programmes, namely; relevance of the objectives: the efficiency with which the programmes have been 
conducted; and the effectiveness of the programmes. On the basis of these evaluations, recommendations 
are made regarding future programmes 
This Executive Summary forms the first part of 'Document 1', which also contains the Commission 
responses to the Panel's recommendations. The Panel's full Report is presented in 'Document 2'. 
This assessment incorporates the final evaluation under the Third Framework Programme. 
At the same time, however, the Panel wishes to draw attention to the two Post-Evaluation Reports 
covering Areas 1&2 (EUR 17457) and Area 3 (EUR 16948) of this programme. These Post-Evaluations 
were carried out in depth, by individuals knowledgeable and experienced in the fields concerned, and 
produced a range of specific Conclusions and Recommendations which call for attention. In taking its own 
overview, the present Panel aims to add a useful general perspective but this should not be regarded as 
repacing or detracting from the important content of the Post-Evaluation Reports. 
3.0 The Panel's Comments and Findings 
3.1 Relevance of Obj ectives 
A wide range of objectives has been set for the IMT programmes; wider than for typical industrial or 
nationally-run programmes. The Panel accepts the general validity and continuing relevance of the 
45 
4. 
objectives, but recognises the difficulty of satisfying all of them simultaneously at project level. In view of 
the quickening pace of globalisation in trade, services, and manufacturing, and the need to strengthen 
European cohesion, optimal operation of the programme requires prioritisation of objectives, and hence of 
project selection criteria, to avoid resources being spread too thinly and focus reduced. 
A more suitable format for the IMT programme in FP5 would be a combination of 'horizontal' generic 
work with a number of selected areas where a more focussed 'vertical' approach is pursued. These focus 
areas might relate to the Task Force interests, though others should also be considered if appropriate. The 
modalities of this programme concept, including the possible implementation of additional RTD 
instruments, need investigation. 
Selection criteria such as scientific and technical excellence, economic and social cohesion, sustainable 
development and quality of life, generic and precompetitive research, transnational participation, effective 
participation of SMEs, and exploitation capability have been generally adhered to. 
3.2 Assessment of Efficiency 
The issue of cost-effectiveness as a measure of the efficiency with which resources have been used in 
pursuit of the programme objectives, relates to both Commission resources and programme participants' 
resources. 
The financial contributions made by the Commission have been generally cost-effective. The funding rules 
have provided an appropriate incentive for participation, while at the same time requiring sufficient 
investment by participants to ensure serious commitment. 
Cost-effectiveness regarding participants' resources depends on the balance between the advantages of 
working collaboratively, with a Commission funding contribution, and the extra management effort 
needed. Generally this balance is positive, but preparation of proposals requires considerable effort, 
averaging about 5% of project cost for each proposal. There is also a high proportion of unselected 
proposals, although indications are that in nearly half of these the research would be done anyway. To 
reduce the total resource cost of proposals, and to avoid deterring capable potential participants from 
applying, the aim should be to improve the success rate to more than 50%. A two-stage procedure, in 
which unattractive proposals are rejected on the basis of a less-detailed first-stage submission, should be 
considered. 
The efficiency of programme management relates to activities in advance of the start of project work as 
well as to management of the work as it progresses. A continuing concern, expressed both by programme 
participants and previous Independent Panels, is the long timescale between close of call for proposals and 
start of work. For both FP3 and FP4 IMT programmes, this ranged between 9 and 15 months. This 
compares poorly with nationally-run programmes, including those of important competitor nations such as 
the USA. This timescale should be reduced to not more than 6 months. Mechanisms to allow work to be 
started with limited contract cover prior to final contract signing should be investigated. 
Management and monitoring of the work under contract has been generally satisfactory, both by the 
Commission and within the project consortia. Positive comments have been received regarding the quality, 
competence and dedication of the Commission Scientific Officers. However, there is evidence that, with 
the expansion of the IMT programme, the greater number of Member States and the increased complexity 
of many proposals, Commission staff are under increasing strain. Programme efficiency is being 
endangered and a review of organisation and procedures is needed. 
The use of external experts to advise and assist Commission staff can benefit the efficiency and 
effectiveness of project management - for example, by providing an independent view , and a healthy 
stimulus, at critical project reviews. However, the Panel wishes to emphasise that for efficiency and 
continuity reasons it is essential that an adequate core of technically competent personnel should be 
maintained within the Commission. 
46 
4. 
The Programme Committees, forming the interface at programme level between the Commission and 
Member States, have played a key role in the planning, formulation and implementation processes. When 
specific funds are allocated to a particular area, eg aeronautics, it is appropriate to involve a panel of 
individuals with specialist knowledge to ensure that the programme continues to develop according to 
market needs. 
The Task Forces have so far had little influence on the IMT Programme. They did not come into existence 
until March 1995, by Which time FP3 was in its closing stages, and FP4 had been formulated and the first 
call for proposals launched. 
In regard to external monitoring, the well-established process of 'Post-Evaluation' provides an overall, 
in-depth perspective on the whole course of a programme, based on documentation, interviews, visits and 
questionnaires. This leads to valuable comments and recommendations for the future, and should be 
continued. 
A useful complement to Post-Evaluation is the newly instituted 'Annual External Monitoring', which 
provides an annual check on progress, thus forming a basis for deciding on adjustments of emphasis, 
procedures, etc. However, the the value of the process would be enhanced if that panel had some access to 
the views of those involved, perhaps by means of a one-day 'workshop' with national representatives and 
project participants. 
In relation to modes of action and tools, while there is certainly an important continuing role for 
precompetitive. generic research, the Panel believes that in view of the increasing intensity of global 
competition, shortening of time-to-market, etc, the Commission should now consider complementing this 
with nearer-to-market RTD but still stopping short of product development. In addition, consideration 
should also be given to RTD invoking Treaty Articles 130 K.L and M and, in selected cases, adding a 
proportion of 'directed' research, strategically planned by or on behalf of the Commission, as a 
complement to the established style of 'invited' research. 
Basic Research projects of industrial relevance, involving research institutions and universities working in 
conjuction with industrial enterprises, have proved to be a very valuable aspect of RTD under the IMT 
programme and should be continued. 
The significance of SMEs within the European industrial scene is well recognised by the Commission and 
is fully endorsed by the Panel. In the IMT programmes, support and encouragement of SMEs has been 
shown by the extensive involvement of already technologically-capable SMEs in industrial RTD projects, 
and by the introduction and development of the CRAFT scheme which supports 'cooperative' research 
commissioned by groups of SMEs which currently lack an internal RTD capablity. 
The participation of technologically-capable SMEs in industrial RTD projects has increased in recent 
years. The involvement with larger companies and with academia across national boundaries has provided 
SMEs with access to new technology options and new commercial opportunities. On the other hand, there 
is evidence that some SMEs are deterred from becoming involved in project proposals because of the long 
timescales and low success rate, or because they fear a possible restriction of their flexibility during the 
course of a project. SMEs are often subject to more dynamic, fast-changing circumstances than large 
organisations. 
The CRAFT scheme has proved to be an excellent means for assisting SMEs with a relatively low 
technology base, to raise their technical capabilities. The two-step evaluation process is very successful. 
The Panel regards CRAFT as a high-priority action, and welcomes the higher funding allocation in FP4. 
The CRAFT action is however demanding in terms of Commission management effort, and more attention 
to the staffing requirement is needed for the future development of CRAFT. Consideration should be given 
to a degree of decentralisation of stimulation to Member States, and the 'bottom-up' approach should be 
maintained. 
47 
4. 
The Panel also wishes to emphasise that the interests and needs of SMEs need to be kept in balance with 
those of larger industrial firms, as both large firms and SMEs have important.contributions to make to the 
EU. 
In regard to Coordination Mechanisms, both the 'Thematic Networks' and the Cluster Concepts' of FP4 
are still in a relatively early state. For fuller exploitation of their potential it will be necessary to go 
beyond merely the coordination of RTD items which arise separately and independently in various 
Commission programme areas. A more proactive, strategic approach will be needed, to set up integrated, 
multidisciplinary groups of projects with forward planning to ensure coverage of all aspects essential for 
commercial exploitation in a competitive timescale. 
3.3 Assessment of Effectiveness 
The Post-Evaluation Reports for FP3 IMT indicate that the initial objectives of promoting technical 
research in all Member States, supporting and encouraging SMEs and the collaboration of industry, 
research institutions and universities across national boundaries, were met to a high degree. The Reports 
also judge positively the achievement of project and programme objectives, including contributing to 
European cohesion and economic development. The number of patents filed averaged less than one per 
project, but patents are not a universal measure of achievement. Much important innovation within IMT is 
not patentable, eg advances in scientific disciplines such as structural analysis, fluid mechanics, etc. The 
considerably larger FP4 Programme is still in its early stages and therefore difficult to judge, but strong 
responses in most programme areas are apparent. 
The overall objectives of the IMT Programmes during the past five years have been met in regard to 
scientific and technical accomplishments. An open question is whether the resulting contributions have 
been adequate to improve European competitiveness, in view of the increasingly strong RTD activities in 
competitor countries. 
The strengthening of the scientific and technological bases has benefitted SMEs, large companies, 
universities and research institutions throughout the EU. In addition to the accumulation of research 
results, it contributes to the forming of collaborative relationships after project completion and, especially 
for SME participants, to the building of confidence for operating at higher technological levels. In this 
respect, participation in the CRAFT scheme has brought particular benefits to SMEs lacking an internal 
RTD capability. 
It is possible to identify considerable contributions to relevant Community policies through the support of 
cooperation across national borders, and the collaboration of SMEs, research institutions and universities. 
Economic and social cohesion has been strengthened. There has been a small direct contribution to 
employment, by recruitment of individuals to work on projects, and a potential created for larger 
employment benefits through industrial exploitation in future. Also, significant scientific/technical results 
have been obtained in areas relating to the environment (recycling, less wasteful manufacture, more 
environmentally-friendly materials, aircraft noise and exhaust emissions,etc) 
In the areas of dissemination and exploitation, a balance needs to be achieved between commercial 
exploitation solely by the project partners who have invested in the work, and the distribution of results for 
wider benefits to the EU. FP3 IMT results have been given controlled dissemination through numerous 
Information Days, Workshops and Conferences. Despite these significant efforts, however, there remains 
a concern that many potential European beneficiaries of the results of the IMT programmes are not yet 
being reached, and more attention needs to be paid by the Commission to dissemination and exploitation. 
Quantitative estimates of exploitation potential in reports produced under Commission contract have 
indicated very high economic returns on programme investments. Comparisons should be made between 
such estimates and the actual benefits achieved, by re-visiting a sample of projects 3 to 5 years after 
project completion. 
48 
4. 
3.4 Importance of the IMT Programme 
The IMT programme is an important and excellent programme. Although the IMT budget in FP4 is 
considerably larger than for FP3, it still represents a small proportion of the total requirement for RTD 
expenditure in this field in Europe. The high application rate of good-quality industrial research projects 
demonstrates the strong willingness of European industry and the research community to participate, and 
points to the need for IMT funding to be further expanded to enable the Commission to respond 
adequately to this willingness, and to the challenge of strengthening the technological competitiveness of 
European industry. 
4.0 Recommendations 
1. The IMT Programme is considered to be an important and excellent programme. It should be 
significantly enhanced within the Framework Programme and very high importance should be allocated 
to the objective of strengthening the technological competitiveness of European industries. The 
financial means and personnel available for the IMT programme are modest for the task in hand and 
should be substantially expanded. 
2. The IMT programme should in future combine a basis of 'horizontal' generic work with a number of 
selected areas where a more focussed 'vertical' approach is pursued. 
3. Measures should be taken to increase the success rate of proposals to higher than 50%. The present low 
success rate is considered to discourage prospective participants from submitting proposals potentially 
important to future European competitiveness. 
4. In order to reduce the cost and staff effort associated with proposal preparation, and the length of the 
processing steps, a two-stage submission procedure is recommended. 
5. To make the IMT programme more efficient and effective, the timescale between proposal submission 
and the start of funding of the work should not exceed 6 months. 
6. SMEs qualified to participate in IMT programmes should be encouraged to apply. Better 
information and communication with SMEs through expanded networks and other means is necessary. 
7. CRAFT is an excellent action in IMT and should receive continued emphasis. Commission staffing 
levels in CRAFT should be increased. Partial decentralisation to Member States of the stimulation of 
CRAFT involvement should also be considered. 
8. Better measures for dissemination of results of IMT projects should be introduced, together with an 
increased focus on the encouragement of exploitation. 
9. Sample groups of IMT projects should be re-evaluated, 3 to 5 years after completion, to compare the 
economic returns actually achieved with the estimates of exploitation potential carried out earlier. 
10.To encourage the beneficial spin-off to other industrial sectors of IMT project results aimed firstly at 
aeronautical applications, it is recommended that a special information/advisory unit be set up to 
promote wider awareness of such advances. 
11.Continued emphasis on the use of universities and research institution expertise, especially in Basic 
Research, is recommended. Basic Research in support of Industrial technologies is considered very 
important in the context of the IMT programme. 
49 
(4) 
4. 
12.The Commission should give consideration, in conjunction with Member States, industry and the 
research community, to the introduction on a selective basis of a wider range of instruments for the 
conduct of IMT RTD activities. 
13.For projects where it is agreed between proposers and the Commission that patents are arelevant 
measure of innovation, the gaining of patents should be included in the objectives. 
14.The Panel commends the two Post-Evaluation Reports (EUR 17457 and EUR 16948), and the 1995 
External Monitoring Report for consideration. 
50 
II. COMMISSION'S RESPONSE 
The Commision is grateful to the Five Year Assessment Panel for their thorough and detailed work 
in examining activities under the IMT programmes during 1991-1995. The Report is welcomed and 
considered fair and balanced. The Panel's positive comments concerning the current programme are 
acknowledged. Continuous improvements are being enacted concerning efficiency of management 
through modern informatics and streamlining of project reporting, financial control and external 
technical assistance. These actions will meet the concerns expressed over limited resources both 
within the CRAFT activity and for the management of conventional RTD consortium projects. 
The efficient management of the programme requires not only a full transparency of procedures and 
commitment of public money, but also the achievement of realistic timescales. The Panel have 
rightly identified these as important. Proposal evaluation procedures are under constant review. To 
achieve gains in time will essentially require accelerating the consultative and administrative 
procedures following proposal evaluation and recommendation, both within the Commission and 
with Member States through the Regulatory Committee. The Panel has identified a recognised need 
to increase SME participation. This is being continuously encouraged and any further move towards 
a more effective decentralized network for stimulating SMEs will ensure a high level of "European 
dimension". 
As projects under the IMT programme and its predecessors mature, exploitation measures become a 
key part of the effectiveness of the programme. The increased attention through the Accompanying 
Measures (and under Activity 3 of Framework IV) is ensuring a significant increase in the visibility 
of exploitation. 
In this context, the "Action Plan for Innovation in Europe" (COM (96) 589) has important messages 
for the take-up of results. Transfers of intellectual property, know-how and experience between 
different industrial sectors are essential. Patents and licensing, additional financial intstruments, the 
effective promotion of technology transfer deals and the encouragement of SME opportunities 
within the broader European context will all become crucial as the millenium is approached. 
The strengths of programmes such as IMT lie in the achievement of appropriate balances between 
basic and applied research on the one hand, and on the other between a generic technical content 
together with selected vertical concentrations on strategic themes. The constant accumulation of 
experience through project selection and management, coupled with feedback from project 
indicators and reviews such as the one just completed by this Five Year Assessment Panel, should 
ensure a continued valuable contribution to Europe's industrial RTD base in the future. 
51 
III. EVALUATION PANEL 
Professor Colin Seabrook 
After an extensive career in engineering and the recipient of a number of awards, Professor Seabrook is 
currently visiting professor in Engineering Design at the University of Bath. He continues to serve on a 
number of U.K. and Institution of Mechanical Engineering Committees and has provided important 
contributions of Advisory Boards within the Department of Trade and Industry and elsewhere within the 
U.K. He was formerly Group Research & Development Director - J. H. Fermer & Co. Ltd. Presently a 
director of CSMA Ltd., he also runs an engineering consultancy. 
Mr. Frits Klostermann 
Trained in physics and mechanical engineering, Mr. Klostermann spent most of the last 32 years of his 
professional career at Philips Research Laboratories, Eindhoven, eventually responsible before his retirement 
for the policy of Philips International Research towards the emerging European cooperative R&D 
programmes. He currently runs a small consultancy company specialising in electronic device physics topics, 
management studies and research evaluation. 
Professor Luis Oro 
Currently professor in chemistry at the University of Zaragoza, Professor Oro has strong links with a broad 
range of R&D topics through his recent position as Secretary General of the national plan for Scientific 
Research and Technological Development in Spain. He has wide experience of OECD Scientific Policy and 
of SME programmes and requirements. Professor Oro is Vice-President of the European Science Foundation. 
His research interests include organometallic chemistry and homogenous catalysis. He has published 
extensively in the field. 
Dr. Jacques Lukasik 
As scientific director of the Lafarge Group (a major construction company operating in 45 countries), Dr. 
Lukasik has considerable experience in the assessment of research projects and interactions with the wider 
scientific and technical community outside his own company. His previous experience includes Research 
Directorships within CNRS, in particular with International and Industrial relations. He has held a series of 
visiting Professorships in the USA, Brazil and Egypt. His technical background in research was originally in 
fundamental atomic and molecular physics, quantum optics, optoelectronics and lasers. 
Dr. Hans Joergen Pedersen 
Currently Vice-President for Corporate Technology and Research at Danfoss in Nordborg, Denmark, Dr. 
Pedersen was a member of the Industrial and Materials Technologies programme review panel for Brite-
Euram II (1991-1994). His presence on the IMT Five Year Assessment Panel provided continuity with this 
earlier review. Dr. Pedersen has had extensive research management experience in his role as chairman of 
the National Committee for the Danish Materials Science Programme, and as a national representative on the 
European Organisation for Testing and Certification (EOTC). Dr. Pedersen's particular research interests are 
in surface coating technology, microsystems and Total Quality Management. 
Dr. Ing. Heinrich Bergmann 
Formerly Director of the Institute for Structural Mechanics at the DLR (German Aerospace Research 
Establishment), Dr. Bergmann has combined his lifetime's experience with aerospace materials and structures 
and general aeronautical engineering with extensive committee advice for National and International R&D 
programmes. He has acted as a consultant to the European Space Agency (ESA) on the HERMES Safety 
Advisory Committee. Dr. Bergmann is Professor of Mechanics at the University of Hannover, visiting 
Professor of Aeronautics at California Institute of Technology and visiting Professor of Engineering 
Mechanics at Virginia Polytech, USA. 
52 
4. 
Mr. Frank Armstrong (Rapporteur) 
Mr. Armstrong, formerly Director Aerospace Vehicles RAE Farnborough and now a consultant, has held 
senior posts in the U.K. Government Aeronautical Research Establishments which involved formulating, 
managing and coordinating National Research and Demonstrator programmes in air vehicle technology. He 
was a member of the review panel for the 1990-1994 IMT programme Area 3 "Aeronautics" and served as 
rapporteur for the IMT Five Year Assessment Panel. 
53 

1996 FIVE-YEAR ASSESSMENT 
STANDARDS» MEASUREMENTS AND TESTING 
55 

L EXECUTIVE SUMMARY 
The specific programmes for standards, measurement and testing in the Second, Third and Fourth EU 
Framework Programmes (FWPs) have undergone major changes. The BCR programme in the second 
FWP was a closed, basically subscription-type programme that became the Measurement and Testing 
(M&T) programme with open calls on specified themes in the third FWP. Finally the Standards, 
Measurement and Testing (SMT) programme in the fourth FWP is a truly open call programme with 
broad themes and a small part covering dedicated calls for urgent needs of the European standardisation 
bodies and the Commission. 
The common objectives of these three special programmes are to strengthen the functioning of the 
European internal market and the competitiveness of the European industry and also to meet the needs of 
society. The themes of the programmes have been enlarged from fabrication of Certified Reference 
Materials (CRMs) and measurement support to Community Policies in the BCR, to measurements for 
Quality European Products, research related to written standards and technical support to trade as well as 
measurements related to the needs of society. This larger scope is also reflected in the budgets that have 
increased from 59.2 MECU in the BCR to 184 MECU in the SMT programme. The funds for the Joint 
Research Centre (JRC) are not included in these figures as the JRC is not included in this evaluation. 
The European added value is very high in the BCR, M&T and SMT programmes as they address 
areas that are essential for the EU to function. The importance of this has grown continuously and is 
today much higher than at the time of the BCR programme. Today the directives and standards that are 
developed are in use and must be enforced. The necessary conformance assessment depends upon 
harmonised and traceable measurements and a uniform and reliable measurement and testing 
infrastructure. In addition there is a huge backlog in the development of standards. All of this needs 
strong support from the specific programmes on standards, measurement and testing. 
The Panel has based its five year evaluation of the BCR/M&T/SMT programmes upon extensive 
interviews, earlier evaluations of parts of the programmes and material provided by the programme staff. 
The resulting recommendations are such that some of them can be implemented almost immediately by the 
programme management, others need decisions by the Commission but could still be implemented during 
the present SMT programme and the remaining ones are intended as inputs to the fifth FWP. 
The role of the programme staff has evolved considerably during the assessment period. The role of 
scientific officers has changed from scientific involvement to a more administrative, consulting and 
monitoring role. This has increased efficiency and more projects are now handled per scientific officer 
than before. This change has at times been painful and the management has not always been able to 
respond to the changing needs. The project teams and members of the Regulatory Committee see the SMT 
programme presently as being well run and the staff as helpful and responsive to needs. The Panel 
recommends that the management of the programme is developed to a more team-based form with 
much delegation and involvement of the whole staff. 
Problems are seen in the slow and complicated general procedures of the FWP in the Commission for 
approving project proposals and letting contracts. The problem is greatest in the dedicated calls where the 
time from proposing a topic to the signing of a resulting project takes at least two years. It is felt that the 
procedures that are created to ensure impartiality and independence may create a false sense of security. 
The Panel recommends that the time from closing a call to signing the contracts for approved 
projects shall be no longer than six months and a special approval process that is no longer than three 
months shall be created for small and short projects for solving urgent European needs. 
The dissemination and exploitation of results of single projects was not a concern in the BCR programme 
as the results, like developed CRMs and intercomparisons, had a clear user group that was easily 
informed. In the subsequent programmes this question needs more attention as the results of the projects 
can be used widely and are from more diverse areas. Despite only 36 of the 173 projects in the M&T 
57 
programme having been completed, the output of projects is already impressive. The Panel recommends 
that the encouraging start to the dissemination and exploitation of the results of the M&T programme 
and in the future the SMT programme shall be continued effectively and more systematically and be 
followed up so that maximum benefits of the project results are obtained. 
The concept of dedicated calls in the SMT programme is seen as very good. The Panel believes it is the 
right concept to provide solutions to specific European needs in standardisation, measurement and testing. 
The Panel recommends that the dedicated calls concept is continued and further developed by 
allocating more funds for this purpose, increasing the number of European groupings that propose topics 
for these calls and speeding up the procedures for approving project proposals for these calls. 
As mentioned earlier, the BCR/M&T/SMT programmes have all had a major significance for the 
implementation of EU policy. The panel sees a strong need, that has been expressed from many 
sources, to keep and strengthen the SMT in the fifth FWP as a separate programme or in some other 
way organise and secure that it remains a strong unit with its own identity and funding. The overall 
objectives of the present SMT programme can be used as the basis for the next programme but some 
revision of focus and objectives as well as additional detailed measurable objectives against which the 
progress of the programme can measured are recommended. 
The panel recommends that the future programme for standards, measurement and testing shall 
include dedicated, special focused and open calls. With this combination, all types of European needs in 
this field can be taken care of. The allocation of funds to different calls and themes should allow for some 
later adjustment, for instance by the Regulatory Committee, based upon the experience of finished calls 
and emerging new needs. Because of the continuing substantial needs in the fields of the programmes it is 
recommended that the total financial resources for the future programme are raised to 300 MECU 
without taking into account the part of the budget allocated to the JRC. The relative share of funds for the 
standards, measurement and testing programme in the FWP V should be clearly higher than at present. 
The production of CRMs has been a valuable contribution to European metrology in the programmes 
evaluated. The panel sees that these activities also need support in the future programme. This should 
be organised within the suggested framework in such a way that the important European needs in the field 
are fulfilled and the high quality of CRMs is secured and maintained. 
The Panel had to do this evaluation with a very tight schedule of only three months. The help of the SMT 
programme staff was very valuable and without it the work could not have been completed according to 
the given schedule. The panel members have all actively contributed to the report and the conclusions and 
recommendations of the panel are unanimous. 
58 
II. COMMISSION'S RESPONSE 
The Commission is very grateful to the Five Year Assessment Panel for its thorough and detailed work in 
examining the activities of the Standards, Measurements and Testing (SMT) programme during the period 
1991-1995. The constructive recommendations made by the panel are welcomed and will be taken into 
account during the 4th Framework Programme as well as during the planning of activities related to 
standards, measurements and testing in the 5th Framework Programme. 
As pointed out by the panel, the SMT programme has undergone fundamental changes in its scope and the 
way that it operates during the last five years. These changes were necessary in order to ensure that the 
programme is managed in the same manner as other specific industrial related research programmes in the 
4th Framework Programme. This evolution is presently continuing towards a form using more delegation 
and team work. 
With respect to other recommendations to be implemented in the short term, the Commission fully 
recognizes the importance of ensuring that calls for proposals are not over subscribed, of reducing delays 
in the project evaluation and selection process, and of ensuring the exploitation and the dissemination of 
the results of projects. Appropriate measures are being taken, and will continue to be taken, in these 
directions. One important development, that will help to increase the exploitation and the dissemination of 
results, is that the progress of projects is now being monitoring against clear quality indicators. The 
Commission's preliminary guidelines for the 5th Framework Programme (Commission documents 
COM(96)332 and COM (96)595), echo the panel's, recommendations concerning the need to focus 
Community research efforts, to increase the flexibility of Community research programmes in order to 
respond to needs as they arise, and to reduce the time taken for project evaluation and selection. 
It is particularly pleasing that the panel has recognized the merits of the dedicated call system in the SMT 
programme. The system, which was largely experimental when it was first set-up, has already evolved 
considerably and will continue to do so in the 4th Framework Programme. It allows Community RTD 
activities linked to standardization to be precisely and efficiently focused where the real urgent needs are, 
and may provide a model for similar systems to be set up in the 5th Framework Programme. 
The Commission fully agrees with the panel on the paramount importance of standards, measurements and 
testing for the future competitiveness of European industry in the global economy and for the general well-
being of society. 
59 
III. EVALUATION PANEL 
Professor Freddy ADAMS (Rapporteur) 
He is currently professor of chemistry at the University of Antwerp where he was until October 1995 
rector and chairman of the University Research Council. He is member of a number of Belgium Research 
Institutions. His main research interest focus on environmental chemistry and material science. 
Mr. Michael BROCK 
Trained in electrotechnic and acoustic engineering, Mr. Brock is Director of the Transducer Products 
Division at Bruël and Kjaer. In addition, he acts as consultant in management and strategical planning of 
technological complex companies. 
Mr. Pierre CROON 
Mr. Croon is currently General Director of the Belgium Institute for Standardization. He is participating 
actively in the European Committee for Standardization (CEN-CENELEC). 
Mr. Gôran LINDHOLM (Chairman) 
Mr. Lindholm is currently Director of NORDTEST, organization for testing in the Nordic countries. His 
previous experience includes directorships at international level in the electronic and information 
technologies area. 
Dr. Walter RAULS 
Currently managing Director of the Standards Commission for testing materials in DIN (German Institut 
for Standardization). He is also participating actively in various CEN working groups. 
Mr. Michel ROUSSEAU 
Trained in mechanical engineering, Mr. Rousseau is auditor for the French Committee for Accreditation 
(COFRAC). 
Dr. James TWEED 
Dr. Tweed is currently technical manager for programmes on testing and design methods at AEA 
Technology Harwell (UK). His technical background in research is in polymers and metal composites. 
60 
1996 FIVE-YEAR ASSESSMENT 
ENVIRONMENT AND CLIMATE 
61 

I. EXECUTIVE SUMMARY 
1.0 Introduction 
The Five Year Assessment Exercise 
This is the Executive Summary of the 1996 Five Year Assessment of Specific Programmes in the Fields 
of Environment and Climate Research and Technological Development (RTD). It covers all Framework 
research activities supported by the Commission in these areas over the period 1991 to mid-1996, 
including activities supported by the Third Framework Programme (1990-94), and all activities included to 
date in the Fourth Framework Programme (1994-98). In compliance with COM (96) 220 final (22 May 
1996), this Five Year Assessment constitutes the final evaluation of the programme launched in the field of 
Environment under the Third Framework Programme. 
The Assessment Panel charged with conducting the Five Year Assessment met on several occasions to 
discuss the background documentation provided by the Commission services; to interview Commission 
staff and representatives of organisations with an interest in environmental research; and to draft the 
Assessment Report. Individual Panel Members held separate discussions with senior members of the 
scientific community across Europe. A questionnaire was also sent by Technopolis Ltd. to all project 
coordinators of the Environment Programme (1990-94) and the Environment and Climate Programme 
(1994-98). In total, 805 questionnaires were distributed and 402 coordinators responded by the cut-off 
date of mid-September (50% response rate). 
Environment and Climate Programmes 
The Specific Programme in the Field of Environment RTD (1990-94) was aimed at contributing to the 
scientific and technical basis for the implementation of EU policy. It was divided into the following areas 
Participation in Global Change Programmes 
Technologies and Engineering for The Environment 
Research on Economic and Social Aspects of Environmental Issues 
Technical and Natural Risks 
Some 554 projects were funded over the course of the programme, which had a project budget of 300 
MECU and an overall programme budget of 315 MECU. 
The Specific Programme in the Field of Environment and Climate RTD (1994-98) has aims similar to its 
predecessor. Amongst other things, it is intended to 
Contribute to world programmes of research into global change 
Satisfy the demands of subsidiarity and contribute to cohesion via the promotion of scientific and 
technological cooperation and integration between European universities, research institutes and 
industry 
Act in concert with the activities of the Member States 
Be of a high quality and strengthen the scientific base in Europe 
Help develop the scientific knowledge and technical competence needed to fulfill environmental 
policy mandates 
Make a significant contribution to growth, competitiveness and employment and encourage the 
participation of SMEs 
The programme has a budget of 567 MECU and is subdivided into four main work areas 
Natural Environment, Environmental Quality and Global Change (47% of the budget) 
Environmental Technologies (25%) 
Space Techniques Applied to Environmental Monitoring and Research (20.5%) 
Human Dimensions of Environmental Change (7.5%) 
63 
6. 
By mid-1996, 313 project awards calling for an EU Contribution of 246 MECU had been granted. 
Another 22 Concerted Actions (EU Contribution of 5.3 MECU) had also been ratified. 
Higher Education Institutes (HEIs) and Research and Technology Organisations (RTOs) dominate the 
programme. For Shared Cost Actions, there were 2,125 successful institutions. These include 1,040 
HEIs; 863 RTOs; 123 firms; and 99 other organisations. The low level of industrial participation is not 
worthy. 
2.0 The Findings of the Assessment Panel 
The Specific Programmes of Research and Technological Development (RTD) in the fields of 
Environment (1990-94) and Environment and Climate (1994-98) have made important contributions to the 
development of environmental science. Programme goals were highly relevant to environmental science 
agendas both in Europe and globally, and the work conducted had the potential to contribute to the 
evolution of environmental policy. Programme execution was for the most part satisfactory, and goal 
attainment in terms of scientific achievement was sound in most areas and impressive in others. There 
was more limited impact on policy development, however, and low industrial participation meant that 
contributions to European competitiveness were marginal. 
Relevance 
The need for a programme of RTD activities in the field of Environment and Climate remains. An 
increasing emphasis on sustainable development will be best served by a highly visible programme of 
support for the European environmental research community which past and present research programmes 
have helped to create. 
The centre piece of this support should be a specific framework programme which combines the current 
Environment and Climate and MAST programmes. Efficient mechanisms are needed to link this with the 
environmental research needs and activities of the Directorate General of the Commission responsible for 
the Environment (DGXI), and with the research needs of other relevant Directorates General. It should 
also be linked with the work of other bodies and agencies such as the European Environment Agency 
(EEA), the European Space Agency (ESA) and the European Meteorological Satellite Organisation 
(EUMETSAT). Improved links with the Joint Research Centre's Environment Institute are needed to 
make the work of this JRC institute more visible to the European environmental science community. 
The goals of the programme are also still appropriate. In particular, there is still a need to strengthen and 
maintain the European scientific base and to conduct policy-relevant research. Ensuring the participation 
of industry is also important to encourage the spread of environmental best-practice as this becomes an 
increasingly important factor in competitiveness. 
Programme compositions were largely in line with programme goals. Environment and Climate projects 
comprised a mix of fundamental and applied research which was relevant to enhanced understanding and, 
to a more limited extent, to policy development. Participants saw projects as a way of expanding 
existingknow-how, creating new European networks and developing new tools and techniques via access 
to complementary sources of expertise in other European organisations. The majority of projects, 
however, were not geared towards enhancing industrial competitiveness. 
Efficiency 
Work within the environment programmes was for the most part adequately financed and conducted in an 
efficient manner. The programme appears to have been conducted in a cost-effective manner. 
Management and administration of the programme was for the most part sound. Some problematic 
management issues should be resolved with the appointment of a new Programme Director; general 
administration was satisfactory; and conduct at the project officer level was commendable. Participants 
were appreciative of the help given by project officials at all stages of the programme life-cycle. Most felt 
64 
6. 
that Commission procedures for making applications were clear and easy to follow, though many still 
regarded them as too slow. Late payment was also a common complaint across the research community. 
There were serious problems in terms of the relationship between the Commission services and the 
Programme Management Committee (COPEC). The transition from an advisory body to a management 
committee has not been smooth or well managed, and efforts will be needed in the future to establish 
productive working relations. 
A number of other administrative problems were also apparent. In isolation none appeared particularly 
severe, but a timely review of administrative functions, procedures and resources, especially staffing 
policies and practices, would help avert problems and maintain current performance levels. 
The range of support modes open to the Commission services (Shared Cost Actions, Concerted Actions 
and Accompanying Measures) was satisfactory, though some aspects of their use were not. Lack of 
support for projects involving large numbers of institutions was regretted. Although these have high 
coordination and communication costs, the benefits of participation often outweigh the costs and a flexible 
approach to their inclusion is recommended. There was also a marked tendency to reduce project funding 
levels with no corresponding reduction in project objectives or ambition. This should be avoided. New 
transparent procedures for assessing project proposals were welcomed, however. 
Effectiveness 
There is little doubt that participants were satisfied with project achievements. Immediate goal realisation 
occurred in terms of networking and financial security, whereas knowledge enhancement was seen as a 
medium-term outcome. 
Project coordinators were convinced that projects had realised their scientific potential and that the 
European environmental science base had been strengthened as a consequence. Networking had helped 
forge a dynamic European research community which was well integrated with national and global 
research agendas and associated communities. 
Some research results had fed into policy discussions, though overall policy impact had been limited. This 
is largely a function of the nature of the work conducted and the complex, lengthy and non-linear ways in 
which scientific evidence feeds into and affects policy environments and decisions. Lack of policy impact 
had been exacerbated, however, by weak links between the DGXII (Science, Research and Development) 
research community and policy-making bodies such as DGXI (Environment) and other Digs concerned 
with environmental issues. 
The networking aspects of the programme had contributed greatly to the cohesion aspects of EU policy, 
though the Less Favoured Regions still won a disproportionately small share of projects. 
The low level of industrial participation in the programme meant that overall contributions to 
competitiveness had been small. In future greater efforts will be needed to encourage industrialists to join 
projects and networks and to realise the industrial potential of projects. 
The overlap between the areas of interest of the Specific Programmes and national programmes is 
sufficient to ensure that the Specific Programmes remain relevant to national needs, but not so extensive 
that they can be considered a substitute for national efforts. 
All the available evidence points to genuine additionality and subsidiarity, with participation in the EU 
Specific Programmes adding an international dimension via access to complementary assets not attainable 
within national boundaries. Many of the projects concerned with global and regional issues could not have 
been undertaken without Commission support. 
Dissemination of the results of project outputs to other participants in the programmes and to the scientific 
community at large had taken place. Dissemination to policy-making 'end-users' and to the industrial 
65 
(5) 
community had been more disappointing. New mechanisms to ensure this occurs need to be put in place. 
This could involve the use of external consultants to advise on dissemination strategies and restructuring 
some future activities into User-focused areas defined in part by relevant user communities. 
3.0 Recommendations of the Assessment Panel 
Continuation 
The Specific Programme of RTD in the field of Environment and Climate should continue into the Fifth 
Framework Programme. Dispersing environment-related RTD activities into other programmes is not a 
viable or desirable option. A critical mass of highly visible environment-related research is needed if 
sustainable development is to be a keynote feature of the Fifth Framework Programme. This does not 
mean, however, that all environment-related RTD activities should be centralised in DGXII(Science, 
Research and Development). Other Directorates General, especially DGXI (Environment), DGVI 
(Agriculture), DGVII (Transport) and DGXVII (Energy) should be encouraged to continue and expand 
their environment-related RTD activities, with improved mechanisms in place to ensure synergy rather 
than duplication. 
Integration 
The Specific Programme of RTD in Marine Science and Technology (MAST) should be integrated into 
an expanded Environment and Climate RTD programme. The arguments in favour of integration out 
weigh those for its continued, separate existence. In particular, there is a need to strengthen the relative 
position of environment-related RTD in order to ensure that sustainable development is central to the Fifth 
Framework programme, and this can best be done by having a critical mass of flagship RTD in a highly 
visible Specific Programme. 
Networking 
The emphasis on networking within the Specific Programme of RTD in the field of Environment and 
Climate should be maintained and strengthened. The benefits of networking across Europe were highly 
prized, readily attained and much appreciated by the environmental research community. These benefits 
are likely to be short lived, however, in the absence of sustained efforts to nurture a culture of 
cooperation, collaboration and networking. The Panel thus recommends a continued focus on network 
building and maintenance between: 
Participants within individual projects 
Participants in different projects 
All interested parties in specific topic areas, especially those requiring interdisciplinary approaches 
Particular efforts should be made to include industry in Environment and Climate networks and projects. 
Segmentation 
The programme structure should segment and differentiate between User-focused and other types of 
research. Introducing policy-driven, customer-oriented User-focused areas would focus programme 
activities and increase the likelihood of policy relevant work having appropriate policy impacts. User-
focused areas should be limited in number; account for no more than 20% of the overall budget; have 
identifiable user communities in environmental and industrial policy-making circles; and a high policy 
relevance in the short- to medium-term. All other proposals should be oriented towards the medium- to 
longer term. Possible User-focused areas might include activities geared towards 
Global issues and concerns, such as uncertainties related to the climate change agenda 
The needs of multi-nation regional systems and regional policy makers, such as regional seas 
issues 
Some of the specific needs.of industry and the industrial policy community, such as research on 
cleaner production technologies 
66 
6. 
Some of the specific needs of DGXI(Environment) related to the implementation of the United 
Nations Framework Convention on Climate Change; waste treatment and disposal technologies; 
and public perception of environmental issues 
Selection 
Selection procedures should reflect the difference between User-focused and other research projects, 
with additional criteria for User-focused projects. In addition to satisfying excellence criteria, proposals 
in User-focused areas should specify policy relevance, potential users and ways in which project results 
might be exploited in policy development, and this information should be taken into consideration in 
selection decisions. Project proposals not targeted at User-focused areas should not be expected to contain 
such detailed information on potential exploitation routes, though its presence would be welcomed given 
that all projects supported within the Specific Programme should be of some relevance - direct or indirect 
- to environmental policy making. 
Shaping 
Individual customer communities for specific User-focused areas (e.g. DGXI (Environment), other 
DGs, national policymakers, industry etc.) should be consulted and involved more extensively in the 
formulation and specification of research agendas and work programmes. This would focus agendas 
more closely on customer needs and would sensitise these user communities to the existence and potential 
utility of research outputs. For the remainder of the programme, the Commission Services should define 
its scope by specifying broad areas of interest and concern to multiple user communities - including the 
scientific community itself. 
Linkage 
Linkages between DGXII (Science, Research and Development) and DGXI (Environment) need to be 
improved, particularly in the policy-relevant User-focused areas. Currently DGXI (Environment) is 
unable to 'see' the results of work supported by DGXII (Science, Research and Development) or to 
appreciate their policy relevance. In part this can be rectified by better 'packaging' of results and 
improved communication channels and mechanisms between the two DGs. Critically, however, DGXI has 
to show more concern and play a greater part in the determination of the research agenda and work 
programme for a specific User-focused area geared towards its short- and medium-term needs as a 
customer of policy-relevant research outputs. In particular, DGXI should make public its medium-term 
work agenda and its implications for research. Similarly, other DGs with an interest in environmental 
issues (e.g. Agriculture, Energy and Transport) should be encouraged to play a greater role in the setting 
of research agendas within DGXII. 
Dissemination 
Dissemination practices should be improved via better packaging and targeting and the customised 
delivery of outputs to user communities. Specifying User-focused areas and involving relevant user 
communities in agenda setting will undoubtedly improve the prospects for the take-up of policy-relevant 
research results. There is still scope, however, for delivery mechanisms to be improved. User 
communities are interested in problem-solving, and thus in information which can help them solve 
problems - whatever its origin. Policy-relevant results of environmental research supported by DGXII 
(Science, Research and Development) therefore have to be packaged together with other data relevant to_ 
specific problems. This then has to be delivered to well specified target audiences via a number of routes, 
e.g. publications, brochures, seminars, workshops etc. DGXII should continue to explore ways of doing 
this, possibly using external consultants to define an effective dissemination strategy. 
Industry 
An attempt should be made to increase industrial participation via a sub-programme oriented towards 
the needs of firms. The evolution and implementation of technologies with reduced environmental 
impacts is crucial to the concept of sustainable development, yet industrial participation is currently low. 
Ways of reducing barriers to entry and making participation more attractive should be sought. A User-
focused area oriented towards some of the needs of industry is advocated. The alternative sign-posting of 
67 
industrial applicants to other Specific Programmes, e.g. Industrial and Materials Technologies, is not 
recommended. 
COPEC 
Attention needs to be paid to the relationship between the Commission services and the Programme 
Committee for the Specific Programme of RTD in the field of Environment and Climate(COPEC). The 
change in status from an advisory body to a management committee has led to a more formal and strained 
working relationship with the Commission services. Action is necessary to restore levels of mutual trust 
and cooperation. The emphasis should be on striking a balance between the relative powers of the 
Commission services and the representatives of national governments. 
Financing 
Pressures to cut individual project funding and spread resources too thinly over a larger number of 
projects should be strongly resisted. Project participants generally receive less than the amounts requested 
in their proposals, with little corresponding reduction in the scope or scale of project objectives. For 
many this involves seeking supplementary funding and an effective reduction of the proportion of the total 
cost covered by the Commission. The need to supplement funds suggests that original requests are not 
overly inflated, and that the scientific integrity of the work could suffer if these requests are not met fully. 
To avoid this happening, the Commission services should scrutinise suggested cuts carefully and resist 
pressures to spread resources too thinly. Access to advice from personnel capable of making realistic 
estimates of project resource requirements, e.g. those with experience of R&D project management, 
should be be considered via the appointment of external 'reference groups'. 
Administrative Procedures and Resources 
A review of administrative procedures and resources should be undertaken across the Commission 
services dealing with Environment and Climate RTD activities. 
Complex bureaucratic procedures lie at the heart of complaints about slow application processes and late 
payments. The Assessment Panel was also told that expanded workloads were not commensurate with 
modest staff increases in the Commission services over the past five years, and that overload problems 
were acute in some programme areas. If this is indeed the case, immediate step child be taken to rectify 
the situation. Programme management has to date been well regarded by the research community, and it 
is important not to let under staffing threaten this situation. 
68 
IL COMMISSION'S RESPONSE 
The Commission welcomes this independent evaluation of EU RTD in environment and climate and will 
take the recommendations into account both in implementing the on-going programme under the Fourth 
Framework Programme and developing the Fifth Framework Programme for the end of the century. 
The Commission accepts that research into environmental issues at the EU level is essential to respond to 
societal needs and aspirations, as well as to address fundamental problems arising from the land, sea and 
air, to assure a safer and healthier planet within the context of sustainable development and growth. For 
this reason, a major theme in the Fifth Framework Programme should be unlocking the resources of the 
living world and the ecosystem in which major problems such as our future water, the environment and 
health, better protection from catastrophes, eco-management and the urban environment will be tackled. 
Networks within and beyond Europe have been an essential element to successful environmental research 
and European teams have been at the cutting edge of addressing our global challenges through their 
contributions to international initiatives such as the Intergovernmental Panel on Climate Change (IPCC), 
the International Geosphere-Biosphere Programme (IGBP), the World Climate Research Programme 
(WCRP) and the International Human Dimensions of Global Environmental Change Programme (HDP). 
These links will continue to be vital in the search for solutions to issues and problems not found or 
concentrated solely in Europe. 
The need to involve the widest possible interest groups in the shape and direction of future environmental 
research is a key element to securing transparency and meeting the citizens' needs. This will continue to 
be a priority for the Commission services in the development and implementation of its research activities. 
Improvements have already been made to strengthen links between the operational research Directorates 
General and the policy making Directorates General to ensure greater synergy and coherency and this 
process will be further developed. 
Commission services recognise that continued efforts need to be made to promote the opportunities for 
industrial participation - both through large and small to medium-sized enterprises - in the research 
activities. These opportunities are being taken up in specific areas of existing research activities such as 
environmental technologies and space, but more work remains to be undertaken to ensure better public -
private sector, and academic-industrial, synergies to maximise the potential opportunities and benefits 
from collaborative environmental research. Such research must never be seen as exclusively for public 
sector organisations or be carried out only through public financing. Nevertheless, the Commission 
considers that the establishment of an industrial subprogramme specifically directed towards the needs of 
industry, as proposed by the Panel, is not the best way forward to assure such synergy or industrial 
involvement. 
Numerous steps are already taken to put EU funded environmental research to the best use. Nevertheless, 
further work will be undertaken by the Commission services to ensure greater awareness and visibility for 
the output of EU environmental research. Solutions need to be found to ensure that the vast array of 
results come in a digestible form as quickly as possible to the attention of those who can best use and need 
them. 
Some brief findings are made by the panel concerning work undertaken by the Environment Institute of 
the Joint Research Centre. Whilst the panel correctly indicates that an assessment of the Environment 
Institute was not within the remit of the panel and has been undertaken by a separate Visiting Group under 
the chairmanship of Professor Carlos A. Borrego, the main report nevertheless refers to expenditure by 
the Environment Institute on environment-related work. These figures are misleading: only 37% of the 
core funding of the Institute (which in 1995 totalled 43.5 MECU and not 46 MECU, as set out in the 
report) is devoted to environmental research, the remaining 63% is allocated for scientific and technical 
support to EU policies developed and implemented by various Commission services (i.e. primarily the 
preparation and implementation of EU directives). The amount expended by the Institute on research 
69 
6. 
during the period of the Fourth Framework: Programme will, therefore, represent 60-64 MECU and not 
half the budget of the whole environment and climate programme (566.5 MECU), as indicated by the 
panel. 
Finally, the Commission recognises in its documents on preliminary guidelines for the Fifth Framework 
Programme that changes need to be made in some of its management techniques and arrangements to 
ensure more efficient management. 
Issues raised in the evaluation report which concern the management of the research activities will be 
taken forward in this context. The Commission services do, however, attach considerable importance to 
the need for constructive and open dialogue with the programme committee, whilst preserving the 
institutional responsibilities for the various actors in the research process, and steps are already being 
initiated to ensure the mutual trust and cooperation essential for the wellbeing of environmental research. 
70 
IIL EVALUATION PANEL 
Dr. Lea Kauppi (Chairman) 
Director General of the Finnish Environment Institute since 1995 
Prof. Niels Busch 
Director of the Danish Russian Energy Research Institute, Moscow 
Former Director of RISO National Laboratory (Denmark) 
Prof. Eloy Garcia Calvo (withdrew due to ill health) 
Professor of meteorology of the Universidad de Alcala, Spain 
Manager of the Spanish national R&D programme on climate 
Prof. Mike Chadwick 
Director (Europe) Leadership pour l'environnement et le développement (LEAD), Switzerland 
Former Director of the Stockholm Environment Institute and Chairman of the 1995 Annual Monitoring 
Panel for the Environment and Climate RTD Programme 
Prof. Wim Harder 
Scientific Director at TNO Institute of Environmental Sciences, Energy Research and Process Innovation, 
The Netherlands 
Prof. André Lebeau 
Chair of the space and space technology programmes, Conservatoire national des arts et métiers, France 
Former President of CNES and Director of Méteo France 
Former Head of ESA and Scientific Director of CNES 
Prof. Francesco Mauro 
Director of the Environmental Department of the National Agency of New Technologies, Energy and 
Environment (ENEA), Italy 
Member of Italian Delegation for the conventions on biological diversity, and desertification and drought. 
Mr. Ken Guy (Rapporteur) 
Director of Technopolis Limited, an innovation policy consultancy, UK 
Former senior fellow at the Science Policy Research Unit (SPRU) at Sussex University. 
71 

1996 FIVE-YEAR ASSESSMENT 
MARINE SCIENCE AND TECHNOLOGY 
73 

I. EXECUTIVE SUMMARY 
BACKGROUND 
Article 4.2 of the Decision 94/804/EC on the specific RTD programme in the field of marine science and 
technology stipulates that the Commission shall have an external assessment conducted by independent 
experts of the activities carried out within the fields/areas/domains covered by this programme, and of their 
management during the five years preceding this assessment. 
This 'Five-year Assessment' covers the relevant activities carried out during the Third Framework 
Programme (from 1991 to 1994) and those carried out during the first part of implementation of the Fourth 
Framework Programme (until early 1996). Where possible, major achievements arising from the Second 
Framework Programme (1987 to 1991) are also reported. This assessment subsumes the final evaluation of 
the MAST-II specific programme under the Third Framework Programme. It also includes an assessment of 
the early stages of M AST-Ill, together with an analysis of the most significant outputs from MAST-I. 
The Assessment Panel consisted of six independent experts who considered the Programme's relevance, 
efficiency, and effectiveness in delivering its objectives, and sought to highlight examples of major 
achievements. The Panel also discussed the requirements for future Community research in marine science 
and technology. 
The Panel undertook personal interviews with:- selected project coordinators; all national delegations on the 
MAST Committee (MAST-COM); representatives from Commission Directorates DGIII, DGXI and 
DGXIV; MAST Programme staff; and the Secretary of the EUROMAR Programme. The Panel also 
examined the results of a standard questionnaire sent to all project coordinators within MAST, as well as a 
wide range of literature relating to the MAST Programme and individual projects. 
The Assessment Panel's report has been prepared in two parts. Part 1 (this document) is the Executive 
Summary containing the main findings and recommendations arising from the Panel's deliberations. Part 2 
(Final Report) describes the development of the MAST Programme; discusses the impact of the Programme 
on a wide range of issues; gives examples of major achievements; presents detailed conclusions and 
recommendations; and includes various statistics related to the Programme. 
DEVELOPMENT OF THE MAST PROGRAMME 
The MAST Programme has developed through MAST-I (1989 to 1991), MAST-II (1991 to 1994) and 
M AST-Ill (1994 to 1998). MAST-I was essentially conceived as a pilot programme (budget 50MECU) 
covering a wide field, with a view to developing a successor programme. Under MAST-II, certain activities 
begun under MAST-I were developed, new topics were introduced, and the geographical coverage was 
expanded. This was summed up in Annex I to the Council Decision of 7 June 1991 (91/351/EEC). MAST-II 
had a final budget of 118MECU, and covered five areas:- (I) Basic and Applied Marine Science; (II) Coastal 
Zone Science and Engineering; (III) Marine Technology; (IV) Supporting Initiatives; and (V) Targeted 
Projects. An area of work included at a late stage was that of Risk Evaluation. 
The M AST-Ill Programme was set out in the Council Decision of 23 November 1994 (94/804/EC), and 
extends, develops and refocuses the activities carried out under the first two MAST Programmes. Its general 
objective is "to foster the scientific knowledge and technological development necessary to understand how 
marine systems function at basin scales, in order to prepare for sustainable use of the oceans consistent with 
the preservation of marine environmental quality and to determine their role in global change." The budget 
for M AST-Ill is 244MECU, and the Programme comprises (A) Marine Science; (B) Strategic Marine 
Research; (C) Marine Technology and (D) Supporting Initiatives. In addition, greater efforts are being made 
to involve SMEs (Small and Medium Enterprises) in the Programme, and stress is being laid on parallel 
initiatives (eg. dissemination/ exploitation/ implementation). 
75 
CONCLUSIONS 
The main findings which the Panel obtained from their assessment are given below under the general 
headings of the effectiveness, efficiency and relevance of the MAST Programme: 
Effectiveness of MAST Programme 
• The MAST Programme has successfully mobilized a European-wide scientific community in marine 
science, through the development of networks associating a large number of laboratories from different 
types of organisations and different countries. The Programme has stimulated interdisciplinary research 
projects which are necessary to achieve the MAST objectives, and it has increased the capacity of 
Europe to participate in large international océanographie programmes. 
• 
• 
• Major breakthroughs have been achieved on a wide range of scientific and (to a lesser extent) 
technological issues. The Panel is convinced that most of these major results could not have been 
achieved without the organisational framework and interdisciplinary approach made possible by the 
funding opportunities within the MAST Programme. MAST is the only one of the various European and 
International programmes which has both the mandate and the budget to encourage European 
collaboration in major research projects. 
The results of the MAST Programme constitute an important scientific and technological asset, which in 
the long term will be used for future technological developments, and for management and policy 
initiatives. The strengthened knowledge base developed in MAST therefore has an intrinsic economic 
value in the long term. 
The development of the code on data management has greatly enhanced the accessibility and value of the 
data collected during all projects in the MAST-II and M AST-Ill Programmes. 
The MAST Programme objectives are relevant to policies in many sectors, including fisheries, 
environment, transportation, industry etc. It is difficult as yet to pinpoint any direct benefits of the 
MAST Programme to national or European maritime policies - indeed most Member States do not 
possess integrated policies on marine affairs. However MAST has strengthened the basic knowledge and 
the scientific expertise required for their development. Within the Commission there seems to be quite 
good liaison between the MAST office and the Fisheries and Environment Directorates, but links with 
the Industry Directorate and the "Maritime systems of the future" Task Force need to be strengthened. 
The MAST Programme seems to have had a significant impact on the coordination between national and 
European marine science R&D policies. In many cases national marine science R&D policies have been 
based largely on experience gained in implementing the MAST Programme. There seems little doubt 
also that the MAST Programme has strengthened cooperation amongst national groups. 
The objective of enhancing European competitiveness in marine technology has only been achieved to a 
limited extent, and the Panel believes that this is due to at least two factors:- the involvement of 
industrial companies in the Programme has been disappointing, and dissemination of the results of 
MAST to the industrial community has been unsatisfactory. 
The Panel welcomes the moves in MAST-III towards the development of better links between research 
workers and the users of their results, whether they be industrial companies or policy makers. 
Within the MAST Programme the "Technology stimulation measures for SMEs" have had a rather poor 
success rate. It seems that either the availability of this supporting initiative from the Commission was 
not disseminated well enough, or that the stimulation measures do not meet the needs of SMEs with 
respect to their R&D activities. 
76 
Efficiency of MAST Programme 
• The Panel was disappointed to learn that very few of the members of MAST-COM came from an 
industrial background, and was also surprised to discover that some members knew little of the MAST 
projects in which their national laboratories were involved, or of the main results of the Programme. 
• The Panel can confirm that the Commission's Programme Officers in the MAST office are highly 
regarded by the MAST community. 
• The publication by the MAST office earlier this year of the booklet "Evaluation of Proposals" has been 
warmly applauded. However the project authorization procedure is still not clear to most of the MAST 
applicants. The Panel also noted that contract negotiation almost always resulted in a budget reduction, 
usually without a corresponding reduction in the project's objectives. 
• The Commission still seems to underestimate the magnitude of the task of coordinating MAST projects, 
but the Panel welcomes the moves in the MAST-III Programme to allow more realistic funding of the 
project co-ordination costs. 
• The Panel very much welcomes the greater attention which is being given to dissemination and 
exploitation in MAST-III, but believes that the supporting initiatives in this respect are still not vigorous 
enough. The Panel suggests that experts or organisations with a broader perspective would often be 
more successful than project partners in disseminating the principal outputs from the MAST projects. 
Relevance of MAST Programme objectives 
• Most of the initial objectives of the MAST Programme (enhancing European competitiveness, developing 
means to exploit ocean resources, establishing the S&T basis for management and protection of the 
marine environment) are still valid because :-
• Europe is facing even more severe competition in marine industries from the USA and Asiatic 
countries than it was ten years ago 
• concerns about over-exploitation of marine living resources are more acute than five years ago 
• possibilities of exploiting deep-sea resources (oil and gas) are now unfolding 
• awareness of the necessity of a better management of European coasts is growing and is becoming a 
critical issue in most countries. 
RECOMMENDATIONS 
The principal recommendations of the Panel are: 
• The European Union must maintain a Programme with a strong component of marine science and 
technology if Europe wishes to exploit the oceans' resources while protecting its marine environment. 
The Programme should include all themes relevant to the marine environment, such as coastal 
management; fisheries and aquaculture; offshore energy; exploitation of marine resources; maritime 
transport; etc. These themes have a considerable overlap in their requirements for science and 
technology, including:- understanding and synthesising natural processes and systems; observation and 
monitoring systems; forecasting systems; exploration and exploitation technology; socio-economic 
modelling; and predicting climate change. 
Extra efforts should be made to involve industry (and SMEs in particular) in the MAST Programme. The 
procedures for proposing, evaluating and awarding contracts for such industrial projects need re­
examining with the purpose of achieving considerable simplification and acceleration, and to make the 
funding arrangements more attractive. 
77 
• Border lines between MAST and EUROMAR should be better defined, and a closer collaboration 
between the two Programmes should be achieved. The panel suggests that projects approved by MAST 
(and those rejected because they are too near-market) should be required to submit a brief project 
description to the EUROMAR Secretariat for possible participation in the EUROMAR Programme. 
• The Panel recommends also that borderlines with other European agencies (such as the European Science 
Foundation and the European Space Agency) need better definition so as to encourage cooperation and 
synergy, and that closer collaboration should be established between MAST and other international 
programmes. 
• Further specific initiatives should be taken to promote dissemination of MAST results, such as (1) 
Production of a regular MAST Newsletter, (2) Organisation of workshops on specific themes between 
scientists and engineers and technicians from industry, (3) Awards of contracts to technical institutes or 
engineering companies to perform a critical review of the relevance and potential uses of the scientific 
and technical results of MAST projects, and to disseminate them under an appropriate format. 
• The MAST Programme should continue to include a mix of large, medium and small projects. However, 
in order to combine the advantages of large and small projects, the Panel recommends that the 
Commission should make better use of the possibility of selecting a suite of related projects (rather than 
one large or very large project), whose partners would be encouraged to exchange ideas and results at 
workshops organised by the Commission under the provisions of 'Accompanying Measures' funding. 
Some recommendations are also made for further improvements in the procedures and management of the 
Programme: 
• For larger projects, the scientific coordinator should be assisted by an executive/administrative 
coordinator appointed and financially supported by the Programme. 
• The Commission should encourage cooperation between European partners to coordinate the use and 
building of large marine research facilities (océanographie vessels, submarine systems, computers, 
hydrodynamic tunnels etc.). 
• The process of selecting projects for funding should be made more transparent, with a full description of 
the criteria used, and of their relative weighting. 
• The Commission should examine very carefully the exact purpose of the contract negotiation stage, and 
having defined its purpose should make this perfectly clear to the project partners. The contract 
negotiation stage should also be made as short as possible. 
• Procedures for 'post-project review' of MAST contracts should be developed, published and 
implemented. 
• The Commission should examine the feasibility of introducing methods to encourage "new blood" 
participation in the MAST Programme. 
Another recommendation refers to the need to achieve much closer cooperation between MAST and the 
Maritime Task Force. 
78 
II. COMMISSION'S RESPONSE 
The Commission welcomes the independent evaluation of the Marine Science and Technology programme 
and will take the recommendations into account when implementing in 1997-98 the second tranche of RTD 
projects of the on-going programme. 
The Commission takes careful note of the argument that a strong component of research on marine science 
and technology must be maintained "if Europe wishes to exploit the ocean's resources while protecting its 
marine environment",. Although the FP5 proposal may not include a dedicated programme on the European 
seas, the various themes highlighted in the evaluation report are likely to be effectively addressed, especially 
through 2 of the priority topics put forward in the Commission document "Inventing tomorrow" which sets 
out preliminary guidelines for FP5 : "Unlocking the resources of the living world and the ecosystem" and 
"Promoting competitive and sustainable growth". 
Attracting more participants from industry has been a long-standing concern for Commission services. The 
second general call for MAST-III proposals (deadline of 15 October 1996) reveals a clear shift towards topics 
in technology, and consequently an increase in the number of industrial participants is expected. More 
efforts on this issue will be made in the future, through focusing of research priorities, more systematic 
dissemination of results of projects, the reinforcement of ad-hoc schemes in favour of SMEs, and continued 
participation in the Task Force "Maritime Systems of the Future". It must be acknowledged nevertheless that 
the very concept of MAST since its creation in 1989, with an important component on studies of processes 
taking place in the sea, precludes strong participation from industry throughout the whole range of the 
programme. On the other hand, the Commission sees great scope to increase the socio-economic relevance 
and visibility of the programme, especially in sensitive areas such as coastal zone management, or the 
monitoring and operational forecasting of coastal seas. 
A special feature of the MAST programme is the existence of regional-scale projects, in the range of 8-12 
MECU of EU contribution. Through these activities, the Commission has experience both in monitoring 
large single projects, centrally coordinated by a prime contractor, and in clustering normal size contracts 
around common objectives. The Commission endorses the panel's recommendation to resort more to the 
latter scheme. 
Linkage with international programmes and organizations is a vital necessity. MAST has already contributed 
to the objectives of i.a. IGBP core projects and intends to take this further. The MAST office is also active 
in EUROMAR and has begun a fruitful collaboration with EMaPS, the secretariat of the European Marine 
and Polar Boards newly set up under the auspices of ESF. 
By implementing a number of supporting initiatives, the Commission has begun to link up European marine 
scientists around common concepts and practices in data management and in modelling. The need to address 
on a significant scale, by other appropriate initiatives, the coordinated use of large research facilities, is 
acknowledged. 
79 
III. EVALUATION PANEL 
Prof. Henk POSTMA (Chairman) 
Background: chemistry. Former positions: professor in physical and chemical oceanology at the University 
of Groningen (1963-86); director (1965-88) of the Netherlands Institute for Sea Research (NIOZ); chairman 
for 10 years of the Netherlands Committee for Sea Research; chairman 1973-75 of the Scientific Committee 
on Océanographie Research (SCOR). Member and former chairman of the Dutch SCOPE-IGBP 
Commission; member of the Dutch Climate Commission. Officer in the Order of Orange-Nassau. 
Mr. Michael W. OWEN (rapporteur) 
Background: engineering science. Chartered engineer. Chairman 1991-95 of the Programme Management 
Committee, coastal engineering research programme, Engineering and Physical Sciences Research Council. 
Member (1992- ) of the Coastal Engineering Advisory Committee, Institution of Civil Engineers. 
Chairman of the steering group of the MAFF research project on joint probabilities of waves, swell, surges 
and tides. 
Dr. Josefina CASTELLVI 
Background: biological sciences. Research professor at "Consejo Superior de Investigaciones Cientificas" 
(CSIC). Main former positions: Vice Secretary General for national coordination of RTD programmes in the 
"Comision Interministerial de Ciencia y Tecnologia" (1995); director of the "Instituto de Ciencias del Mar", 
CSIC (1994); Spanish representative in the Antarctic Treaty (1988); manager of the National Programme of 
research in Antarctica (1989-1994). 
Dr. Kari KVESETH 
Background: environmental chemistry. Since 1993: director of the Division of Science and Technology, 
Research Council of Norway. Chairperson of the Council, Norwegian Space Centre. Member of the Board, 
Norwegian Academy for Technical Sciences. Member of the Armed Forces Council for research policy. 
Main previous honorary positions: chairperson of the Board, Institute of marine research; member of the 
National Committee for environmental research. 
Prof. Pierre PAPON 
Background: physics, applied mathematics. Professor of physics at Ecole Supérieure de Physique et Chimie 
Industrielles (ESPCI), Paris. Chairman of the board of Observatoire des Sciences et Techniques (OST) since 
1990. Member of the Executive Council (since 1992) and of the Bureau (since 1995) of the European 
Science Foundation. 1989-95: president-director general of IFREMER. 1982-86: director general of CNRS. 
Former board member of Agence Française pour la Maîtrise de l'Energie, Commissariat à l'Energie 
Atomique, Compagnie Générale d'Electricité. 
Mr. Johannes POST 
Background: oceanography, and also: geophysics, meteorology, theoretical physics, applied mathematics. 
Experience in off-shore and deep-sea surveying, computer simulation models and computer software for 
marine applications. Founding member and managing director of Hydromod Service GmbH, Hannover, 
since 1989. Founding member of OSAT GmbH, Heist (1989). Board member of the GEOMAR Enterprise 
Association. 
80 
1996 FTVE-YEAR ASSESSMENT 
BIOTECHNOLOGY 
81 

I. EXECUTIVE SUMMARY 
Introduction 
The period of this evaluation is the five years from mid 1991 to mid 1996. The framework II BRIDGE 
programme covered the period from 1990 to 1993 with an expenditure of lOOMECU. Biotech-I in 
framework III extended from 1992 to 1994 and had a budget of 186MECU. The framework IV Biotech-II 
programme extends from 1994 to 1998 with a budget of 588MECU. Thus the period under review covers 
the end of the BRIDGE Programme, all of Biotech-I and the first part of Biotech-II. The BRIDGE 
Programme has been evaluated separately and a comprehensive report is available. The BRIDGE report is 
used as the first frame of reference and changes that occurred in Biotech-I are commented on. It is too 
early to reach definitive conclusions about Biotech-II. Thus this evaluation has concentrated mainly on the 
activities of Biotech-I but with some comments about changes that are noted in Biotech-II. It incorporates 
the final evaluation under the Third Framework Programme. 
The main findings of the evaluation 
1. The initial objectives of the programme remain valid in spite of rapid changes that have taken place in 
the science and technology. Close attention to the relevance of the programmes needs to continue since 
Molecular Genetics and its associated disciplines now provide basic tools in almost all the bio-industries. 
It is important to strike a balance in the objectives between the long term commitment required to realise 
the benefits and the need for flexibility in response to the advance of the basic science and technology or 
the demands of special situations. 
2. Whilst remaining true to its initial objectives, the programme has demonstrated the flexibility to change 
and has therefore evolved over time. The Panel approves of this ability to change but new targets, or 
changes of emphasis within existing targets, need to be publicised earlier and more widely among 
scientists in the community so that they can be aware of how their own field of expertise matches the 
evolving priorities of the programme areas. 
3. One change that the Panel noted was that Biotech-I was oriented more towards basic research compared 
to the earlier and following programmes. Whether due to this or some other cause there was a lower 
participation by industry in that programme. 
4. The selection criteria are followed in the main and continue to contribute to the building of a European 
scientific community although it appears that the cohesion criterion is not so obviously used. There is now 
little or no evidence of resistance to trans-national collaboration among scientists. However it is still the 
case that many projects are not truly interdependent being more a collection of largely independent sub-
projects. While such projects do frequently contribute to skill and technology transfer between partners, 
the opportunity to deepen further the relationships between European laboratories may be being missed. 
5. With the parameters within which it operates, the programme is highly cost effective, having an impact 
far greater than its expenditure would predict when compared with the total European effort in this field. 
To achieve this it relies on the infrastructure and basic science support of the National programmes. 
6. The performance of the Scientific Officers continues to receive favourable comments but there are a 
number of areas in which the management process could be improved. These are discussed and 
recommendations made. Of particular importance, however, is the delay in starting projects after the 
initial funding decisions are made since this can affect the competitiveness of industry and may inhibit 
industrial participation. 
7. As the programmes have enlarged, the role of the Programme Committee has changed. There is less 
clarity in the relationship of the Committee to the evaluating panels. Although the Committee approves 
policy, its role in implementation has become more passive. While there is no case for it to become 
83 
involved in evaluation there is a need to ensure that the committee can monitor the link between policy and 
implementation. 
8. The short term performance of the programme assessed by its publication record seems to be good, 
even though all publications may not yet have appeared. However, real time analysis of the progress of 
each project against its own objectives would be a valuable addition to performance evaluation and new 
project selection criteria. Effects on industrial competitiveness and results in the market will not be seen 
until some time after the end of a programme. It will be necessary to evaluate these features on a much 
longer time frame. However, it is already clear that the number of patents filed by projects in Biotech-I is 
very low. 
9. The Panel has found that some companies have difficulties with the rules of ownership of the results of 
projects that may have an impact on exploitation. Since this may not be a universal problem an 
investigation is needed to assess its potential impact and guide the development of procedures that will 
eliminate it. 
10. The Panel has found evidence of a need for a broader consultation with industry in the formulation of 
programme objectives. In particular, the high level advice provided by IRDAC, the SAGB and the major 
trade associations may not adequately represent the experience of small companies nor of the scientific 
middle management of large companies. 
11. The Panel has noted that there are differences in the needs and methods of operation of small and large 
companies. The structure of the present programmes seems to support the needs of large companies well 
since they can take a longer term view. Small firms need mechanisms that support their nearer-market 
approach. Separate instruments that address their knowledge requirements more directly should be 
investigated. 
12. As a result of its own experience, the Panel is very concerned about the inefficiency of the procedure 
for external evaluation. The period preceding the appointment of the Panel was subject to very extensive 
administrative delays that left the Panel a mere three months in which to perform its task. This means that 
the analysis could not be based on independent data collected at the instigation of the Panel and at its 
worst, could call into question value of the whole exercise. 
13. The Panel has recognised a number of major achievements of the programme such as the Lactic Acid 
Bacteria project, the Lipase project and the genome sequencing and structure projects, particularly the 
Yeast Genome sequencing project. It is noticeable that some projects that are leading to industrial 
application require support in more than one programme to achieve success. The mechanism of the 
continuation of support through competitive bids of a project that has already been approved and is a 
success, may need to be reviewed. The real time monitoring of projects in progress could be of 
importance in this respect. 
14. The Panel has noted that there is very little co-ordination between the EU and National programmes in 
biotechnology, and even less between the various programmes of the Commission. Some duplication or 
overlap between Community and National programmes is unavoidable. It even has some advantages and 
it may be essential if Community programmes continue to build on the infrastructure and basic scientific 
capabilities resulting from the National programmes. On the other hand, the lack of co-ordination 
between the biotechnology components of the various EU programmes is inefficient and wasteful of 
resources. The best way to overcome this is to combine the biotechnology aspects of all three current 
programmes (FAIR, BIOMED and Biotech-II). 
15. The EU programmes are continuing to support the scientific base in that a real scientific community 
has emerged and continues to be strengthened. The programmes also add value to the European research 
effort through projects that could not be co-ordinated effectively by National programmes alone. 
Principal recommendations 
84 
The Panel has made a set of recommendations under five headings that are set out in chapter IV of this 
document . The ten principal recommendations from this set are given below. 
1. Since there is evidence that the distribution of biotechnology topics between three programmes 
(Biotech-II, Agriculture and Fisheries and BIOMED-II) is resulting in a loss of synergy we recommend 
that the biotechnology aspects of the current programmes be combined in future programmes but with 
clear linkages to the user sectors concerned. 
2. As a means of addressing the special problems of small companies and to stimulate processes that can 
lead to the initiation of more start-up companies, we recommend the development of financial platforms, 
analogous to or alongside the present industrial platforms, with the aim of channelling private sector 
funding into the exploitation of opportunities generated by EU programmes. 
3. The Panel is of the opinion that the procedures for consulting with industry are too narrow and at too 
high a managerial level. We recommend that the consultation process should be modified to take into 
account the views of middle management industrial scientists, the views of users and of the managers from 
small companies. 
4. The Panel confirms its agreement with the conclusion of previous evaluations that training is one of the 
most important features of the programmes. We note that while the training component of Biotech-I was 
not improved compared to BRIDGE, that of Biotech-II up to the present, is better. We recommend that 
the training activity of Biotech-II be carefully monitored to maintain this improved performance. 
5. The Panel considers the process of project evaluation and selection to be fair and supports the strong 
adherence to peer review and the ranking of projects by scientific quality. The Panel makes a number of 
recommendations to improve the process based on thé comments of scientists that have been directly 
involved. Of these, probably the most important is that greater emphasis should be given to the quality of 
the feedback notes. This is one of the main contacts between Commission staff and the scientific 
community and it can be of immense value in the preparation of future applications. 
6. The Panel have repeated the observation of earlier panels that the internal Commission procedures for 
approving project selection following scientific peer review evaluation are unacceptably long and severely 
reduce the competitive impact of the programme. We recommend that a radical review of procedures 
should be undertaken to eliminate delays and to reduce the administrative burden on Commission staff. 
7. The Panel has observed that projects that support infrastructure activities, such as databases and culture 
collections, suffer from difficulties due to the rigid contractual arrangements of the Commission and the 
need for the continual renewal of short term contracts for the support of a long term function. We 
recommend that a new mechanism be devised to provide long term contracts for infrastructure services 
including built in reviewing procedures to check their performance as a justification for the continuation of 
funding. 
8. The Panel has received evidence that the Programme Committee has become remote from the process^ 
of project selection and that this separation may permit inadvertent changes of policy due to the spread of 
projects selected. We recommend that the Programme Committee be given a quality assurance role in the 
process of project selection so that the link between policy and practice is fully monitored by the 
Committee. 
9. As a result of its experience in the initiation of this evaluation and because of its knowledge of the 
preceding administrative delays, the Panel is seriously concerned that the actual evaluation of programmes 
is not given proper priority. We recommend an independent, in depth review of the external evaluation 
process compared to the Commission objectives, to include the difficulties experienced by this Panel as a 
The full report of the panel can be obtained separately. 
85 
result of the implementation of the current exercise and the use that could be made of real time monitoring 
of projects in programme evaluation. 
10. The present practice leaves evaluation panels, and perhaps others, unaware of the use that is made of 
the results of their efforts. We recommend that the Commission should make an explicit public response 
to the recommendations made by external evaluating panels with reasons for their acceptance or rejection. 
86 
II. COMMISSION'S RESPONSE 
The Commission wishes to express its appreciation to the panel.for its report on the assessment of the 
activities of the Biotechnology Programmes for the last five years (1991-1995). The Commission 
recognises the fact that the report was carried out under a pressure of time, however, it regards the 
outcome as fair, accurate and as having reached balanced conclusions and important recommendations. 
The Commission will comment on the main recommendations: 
- As the panel recommends, the biotechnology aspects of current programmes should be combined in 
future programmes. The future programme covering Life Sciences and the Ecosystem can focus on the 
opportunities clarified in the past, so as to reap the benefits of a concentrated effort combining all 
resources and means which may be obtained in a single unified programme. 
- In addressing the special problems of SMEs, a set of 'stimulation measures for SMEs' has already been 
introduced. There seems to be a general appreciation for these measures and the way in which they are 
implemented in the Biotechnology programme. A continuous evaluation and further development of these 
measures is needed in order to optimise them. 
- Concerning industry, the Commission receives advice from IRDAC (Industrial Research and 
Development Advisory Committee), especially for the preparation of the Framework Programmes. In 
addition to this institutional and strategic advice provided by IRDAC, the Commission services receive 
specific advice from other industrial groups throughout the crucial steps of programme preparation and 
implementation. 
- As far as training is concerned, the Commission is making every effort to maintain the current success 
rates which are the result of the recommendations made by the evaluation of BRIDGE. In addition, we are 
devising methods for monitoring the quality and outcome of the increased number of grants. 
- The Panel considers the project evaluation and selection to be fair. In addition it gives great emphasis on 
the quality of the summary consensus reports which serve as a feedback communication mechanism to the 
applicants. The Commission takes this remark very seriously. Currently, the quality of the summary 
consensus reports have significantly improved compared to the reports of the previous call for proposals. 
The importance of adequate feed-back to proposers is emphasised to the external evaluators participating 
in the different peer-review panels. 
- Since one of the main goals of the Framework Programme is to enhance the competitiveness of European 
Industry, measures are being taken in order to increase the transfer of successful research results into 
marketable products. The involvement of industrial experts, as well as experts from the industrial 
platforms, in the preparation of the programme and in the selection of projects, constitute means for the 
promotion of the industrial use of research results. 
- The Panel made the observation that the internal Commission procedures concerning project evaluation 
and selection are rather long. The real delays for the Commission's decision after peer reviews are 
approximately two and a half months. The Commission is making a great effort to reduce it further, but 
without detriment to the quality and fairness of the work. 
- The Panel suggested the improvement of the coordination among EU and national programmes in 
biotechnology. The Commission has already taken measures to do this, aiming at increasing programme 
efficiency, synergy and added value. An example is the pilot study dealing with protein engineering and 
structural biology activities in Europe, conducted by the Biotechnology programme. 
- The Panel has received evidence that the Programme Committee has become remote from the process of 
project selection. According to the Commission, there is no reason to believe that the role of the 
committee has changed. Indeed, Article 7 of the Council decision stipulates what the committee's role 
87 
consists of. This has not significantly changed over time, except for the threshold of financial commitment 
under which the committee's opinion is not required (now 0.5 MECU, 0.3 MECU before). The 
Committees' opinion is now also sought on training grants and all accompanying measures. 
- The Panel is seriously concerned that the actual evaluation of Programmes is not given proper priority. 
This statement is based on the fact that the administrative delays are by far longer than the actual time 
given to the Panel to finalise their exercise. The Commission acknowledges and regrets the time lost 
before the panel could carry out its work. It is to be expected however that further assessment and 
monitoring exercises will benefit from the fact that monitoring and assessment have now become a 
systematic part of the daily routine of the Commission's tasks in the framework of the implementation of 
RTD programmes. 
- The Panel suggests that the Commission should make an explicit public response to their 
recommendations. The present note serves exactly as that; a response to the recommendations made by the 
Panel. The Commission finds these recommendations valuable and timely, since the debate and reflection 
on the Fifth Framework Programme has started. 
88 
III. EVALUATION PANEL 
Prof. Dr. Rudolf CASPER 
Prof. Casper participated in the Panel for Monitoring 1995 for the Biotechnology Programme. He was the 
Director of the Institute of Biochemistry and Plant Virology of the Federal Biological Research Centre for 
Agriculture and Forestry, Braunschweig, Germany until 1995. This institute is responsible for the 
evaluation and admission of field experiments with transgenic organisms in Germany. 
Dr Luke Gregory GEORGIOU 
Dr Georgiou is the Executive Director of PREST and has experience in evaluations, science policy and 
technological innovation. He is a founder member of Commonwealth Consultative Group on Technology 
Management and consultant to the UK Technology Policy. 
Dr Marti LEISOLA 
Dr Leisola is the Technology Director of process technology at the Technology Development Centre, 
which coordinates and finances applied, technical and industrial R & D in Finland. "He has experience in 
evaluating national and international R&D programmes. 
M. Paskal BYE 
Mr. Byé, Director of Research at INRA (Institut National de Recherche Agronomique) participated in the 
evaluation of the French "Biotechnologies" programme which covered a 10 year period (1982-92). This 
evaluation was organized by the CNER (Comité National d'Evaluation de la Recherche). 
Prof. Giampiero RAMPONI 
Prof. Ramponi is a Director of the Institute of Biological Chemistry of the University of Florence. He is a 
member of the progress and evaluation for the group on Biotechnology and Bioinstrumentation of the 
CNR. 
Prof. Luis NAVARRO 
Prof. L. Navarro has been the manager of the National Programme for Agricultural Research in the 
Interministerial Commission for Science and Technology (CICYT). He has collaborated with FAO, the 
Hunan Horticultural Institute in China and the Agriculture Ministry of Cuba. 
Prof. Gûnther KREIL 
As well as his academic activities, Prof. Kreil has served in the EMBL Council, the EMBO fellowship 
committee at the Max Planck Institute for Biochemistry (Munich) and for Experimental Medicine 
(Gôttingen). 
89 

1996 FIVE-YEAR ASSESSMENT 
BIOMEDIÇINE AND HEALTH 
91 

T. FmirnvF. SUMMARY 
We have reviewed the last five years of the European Union Biomedicine and Health Research Programme. 
The Review encompasses the three calls of Biomed 1, under the Third Framework Programme1, and the first 
call of Biomed 2 under the Fourth Framework Programme. Our assessment of the Programme and our 
conclusions and recommendations are based on data provided by the CEC, various monitoring reports of the 
Biomed 1 and Biomed 2 Programmes and the analysis of data generated by a questionnaire which attracted a 
response of 44% from Biomed 1 co-ordinators and 25% from Biomed 2 co-ordinators. 
The Research Topics of Biomed 1 were: Pharmaceuticals; Occupational and Environmental Health; 
Biomedical Technology; Public Health/Health Service Research; AIDS, TB, Infectious Diseases; Cancer 
Research; Cardiovascular Research; Brain Research; Chronic Diseases, Ageing and Rare Diseases; Human 
Genome Research; Biomedical Ethics. In Biomed 2 the topics were similar with the addition of 'Legal 
aspects and horizontal activities' including demonstration projects as a separate topic. 
Both programmes attracted a large number of applicants; 1640 in the case of Biomed 1 of which 403 (25%) 
proposals were funded and 1709 in the first call of Biomed 2 of which 302 (18%) were funded. In Biomed 1 
the largest number of proposals were submitted in the Research Areas of Brain Research, Cancer Research, 
Cardiovascular Research, Chronic and Rare Diseases and Ageing, and Human Genome Research. The 
proportion of applications to Brain Research, Chronic and Rare Diseases and Ageing and Public 
Health/Health Service Research was increased in Biomed 2 relative to Biomed 1. In Biomed 1, concerted 
action (CA) was the main implementation mode, the only exception being Human Genome Research in which 
the CEC did fund shared cost (SC) proposals. In Biomed 2 the predominant mode of implementation was 
SC. 
The total sum requested in Biomed 1 was MECU 824 and the amount awarded was MECU 134. In Biomed 
2, MECU 317 were requested and MECU 146 were awarded. This reflects an increase from 16% to 46% 
awarded relative to that requested in Biomed 1 compared with Biomed 2, respectively. The average EC 
contributions per contract in Biomed 1 and Biomed 2 (first call), respectively, were KECU 304 and 350 for 
CA and 585 and 614 for SC. 
We conclude that the Biomedicine Programme (Biomed 1 and First Call of Biomed 2) has achieved the 
objective of contributing significantly to the improvement of medical and health research and development in 
Europe by facilitating the establishment of new collaborations and/or consolidating and strengthening existing 
collaborations. The Programme has satisfied the criteria of 'European added value', 'European dimension', 
and 'subsidiarity' and is characterised by a multidisciplinary as well as multinational approach which makes it 
distinct from Biomedical programmes supported by National Research Bodies and Industry. 
The output of the projects supported by the Programme, as assessed by the analysis of Biomed 1, would 
appear to be qualitatively and quantitatively equal to or better than that of many projects supported by top 
National Funding Bodies. The shift from concertation between many partners in Biomed 1 to actual 
research projects between smaller groups of complementary partners in Biomed 2 is a highly positive trend 
for which the CEC is to be commended. 
The Biomed Programme has also established links with other Programmes of the European Union; 
integration with PECO and INCO3 has promoted participation of non-Member States from Central and 
Eastern Europe. 
Notwithstanding the success of the Programme, we are concerned by the relatively massive oversubscription 
and therefore wastage, due to the limited budget, of a large amount of European talent and expertise. 
This 5-year assessment incorporates the final evaluation of the Biomedicine and Health Programme under the Third Framework Programme 
Pays de l'Europe centrale et orientale (Central and Eastern European Countries) 
Specific Programme for International Co-operation (in the 4th Framework Programme) 
93 
The method of selection of proposals in Biomed 2, based on highly efficient and effective scientific peer 
review, is a marked improvement on the methods of selection used in Biomed 1. Nonetheless further 
improvement is possible, and we therefore endorse the 1995 Biomed 2 External Monitoring Report which 
recommended that (i) unless there are exceptionally compelling circumstances, proposals should be funded in 
strict rank order irrespective of whether or not subareas were represented, and (ii) that the funding of 
proposals should be guided by scientific advice and follow a stepwise or sliding scale proportional to the rank 
order of the final score of the proposal. 
We are sensitive to the need for the strategic, national and political factors that may of necessity influence the 
selection of proposals in various areas. A balance could be achieved between the funding of 'response mode' 
versus 'strategic top-down' driven research if some degree of science-led flexibility (say 10%) was introduced 
in determining the size of the financial envelopes that are set around the various Research Topic Areas by the 
CEC and the Member States. Nonetheless, in order to maintain the apparent high quality of the outcome of 
the Biomed Programme, and to ensure equity at a time when research funds are exceedingly scarce, we 
recommend that irrespective of the strategic importance of any given area, the CEC should establish a 
minimum score below which no proposal can be funded. In this respect we commend the Commission for 
implementing adequate monitoring systems in the form of the Project Review Board and external monitoring 
panels. 
Five aspects of the Programme deserve special attention. These are: (i) the relatively small proportion of 
industrial contractors; (ii) the relatively small number of proposals concerned with research on 'Illicit Drugs'; 
(iii) the long delay between the receipt of proposals and the transfer of funds (median of 12 to 16 months); 
(iv) the few female expert evaluators and members of monitoring panels; (v) the shortage of CEC Staff 
involved in the Biomedicine Programme. Analysis of these issues and recommendations of how any 
deficiencies might be rectified are discussed in the text and in the Conclusions and Recommendations. 
We highly commend the EU for establishing the Biomedicine Programme which has a major impact on 
European Biomedical Research, and will have an important positive effect on the prevention, diagnosis, 
management and treatment of diseases in Europe. The Programme provides the EU with an incredibly 
powerful network and database of unrivalled expertise in Medicine from molecules through to man and 
Society, which in addition to fostering highly productive and integrated research between the Members 
States, serves as a vehicle for the rapid deployment of European Task forces to deal with strategic, and/or 
national issues as they arise. We very much hope that the Biomedicine Programmes of the Third and Fourth 
Framework will form the platform for a similar programme in the Fifth Framework, and that such a 
Programme is allocated a significantly larger budget in order to reduce the wastage of many excellent 
initiatives which reflect the Biomedical power of the Member States. 
Conclusion and Recommendations 
(I) We conclude that the Biomed Programme (Biomed 1 and First Call of Biomed 2) has achieved the 
objective of contributing to the improvement of medical and health research and development in 
Europe by facilitating the establishment of new collaborations and/or consolidating and strengthening 
existing collaborations in Europe. The Programme has succeeded in satisfying the criteria of 
'European added value', 'European dimension', and 'subsidiarity' and is characterised by a 
multidisciplinary as well as multinational approach. 
(ii) As a consequence, the research supported by the EU Biomedicine Programme is quite distinct from 
the Biomedical programmes supported by National Research Bodies and Industry. 
(iii) In addition to promoting the formation of new and major European Biomedical consortia, the output 
of the projects supported by the Programme, as assessed by the analysis of Biomed 1, appear to be of 
a quality and quantity equal to or greater than those of many projects supported by National Bodies. 
This assertion is based on (i) the positive nature of the new links forged between partners, including 
industrial partners, (ii) contributions to novel methods in health care, (iii) the relatively high number 
94 
of patents that have been filed, granted and licensed (i.e. contributions to wealth creation), and (iv) the 
high impact of the publications produced by a large proportion of the projects supported by the 
Biomed Programme. 
(iv) As a consequence of CEC policy and a substantial increase in budget, there was a major shift from 
Concerted Actions and Centralised Facilities in Biomed 1 to Shared-Cost projects as the predominant 
implementation mode in Biomed 2. This shift from concertation between many partners to actual 
research projects between a smaller group of complementary partners is a highly positive trend for 
which the CEC is to be commended. 
(v) Notwithstanding this positive trend, we are concerned by the fact that the relatively limited budget of 
the Programme (albeit increasing over the years) meant that only a minority of proposals could be 
funded in Biomed 1, and even less in Biomed 2. That is, the level of oversubscription was relatively 
high with less than 15% of proposals funded (Biomed 2, first call) in Biomedical Technology, Brain 
Research and Cancer Research, and a mean of only 18% of proposals funded for the Programme 
overall. Furthermore, the level of funding for each selected proposal is variable, and in some cases 
might not allow achievement of all the original objectives that had served as a basis for selection of the 
projects. The apparent positive outcome of Biomed 1 (see 3 above) suggests that for CAs, at least, 
negotiations between the CEC and the contractors led to a formula whereby CEC together with 
National or other (e.g. Industrial) contributions ensured the success of the research proposed. The 
outcome of Biomed 2, in which SC is the predominant mode of implementation, has yet to be 
determined. 
(vi) The Biomed Programme has also established links with other Programmes of the European Union; 
integration with PECO and INCO has favoured participation of non-Member States from Central and 
Eastern Europe. 
(vii) Monitoring systems are in place in the form of the Project Review Board which reviews annually the 
progress of each project and has the power to implement changes or terminate unsatisfactory projects 
on the basis of written submissions and, if necessary, site visits. The CEC has also complied with the 
EU requirements for detailed monitoring of the Programme as a whole of which the present and the 
1995 External Monitoring Report of the First Call of Biomed 2 are examples. Nonetheless, there is 
room for improvement in that monitoring could be more rigorous, systematic and accessible. We 
recommend that the PRB should continue to monitor the scientific progress of the Programme, but in 
addition should also monitor the impact of the funding of the Proposals and Areas with respect to the 
Output of the Programme. The administrative monitoring of the progress of contracts should be 
carried out by dedicated CEC personnel using computerised databases. This would facilitate more 
direct and rapid evaluation of changes which are made during the Programme, alert the CEC to the 
need for additional changes and facilitate greatly the work of the external monitoring panels. 
(viii) The method of selection of proposals in Biomed 2 is a marked improvement on the methods used in 
Biomed 1. We strongly commend the CEC for implementing the new Biomed 2 selection method 
which entails highly efficient and effective peer review and at the same time secures contractual 
confidentiality and secrecy. The latter is important for industrial participation and, by providing 
proponents with an assurance of confidentiality, may have played a significant role in the continued 
high response rate to the call for proposals in Biomed 2. 
(ix) We endorse the 1995 Biomed 2 External Monitoring Report which recommended that (a) unless 
there are exceptionally compelling circumstances, proposals should be funded in strict rank order 
irrespective of whether or not subareas are represented, and (b) that the funding of proposals should 
be guided by scientific advice and follow a stepwise or sliding scale proportional to the rank order of 
the scores of the proposals. It remains to be seen whether these two principles are implemented for 
the second and third calls of Biomed 2. 
95 
(x) Notwithstanding this recommendation (ix) we are sensitive to the need for the strategic, national and 
political factors that may of necessity influence the selection of proposals in various areas depending 
upon (a) the special needs of the EU, (b) the occurrence of apparently new diseases (e.g. AIDS), (c) 
the recrudescence of known diseases (e.g. tuberculosis), (d) the emergence or prominence of known 
diseases due to changes in demography (e.g. Alzheimer's Type Dementia and other age-related 
disorders) and/or (e) novel developments which may promote health care or wealth creation. We, 
therefore, recommend that the allocation of funds to Research Areas should ensure that the best 
science is funded whenever possible and at the same time allow strategic considerations to be taken 
into account. A balance could be achieved between the funding of 'strategic top-down' versus 
'response mode' driven research if some degree of science-led flexibility (say 10%) was introduced in 
determining the size of the financial envelopes that are set around the various Research Topic Areas. 
The size of the financial envelopes would still be determined by the CEC and the Member States on 
the basis of strategic considerations, but could be adjusted by the Programme Committee in response 
to the quality of the proposals in the different Research Areas . This would allow a degree of freedom 
in shifting funds between Areas to facilitate changes in emphasis or success in European Biomedical 
Research, while at the same time ensure that those Biomedical Areas of greatest strategic importance 
for the Member States would be represented in any particular call for research proposals. 
(xi) However, in order to maintain the apparent high quality of the outcome of the Biomed Programme, 
and to ensure equity at a time when research funds are exceedingly scarce, we strongly recommend 
that irrespective of the strategic importance of any given area, the CEC should establish a minimum 
score below which no proposal can be funded. Using the present Biomed 2 scoring system (0 to 4, 
where 4 is excellent) and the analysis of the first call of Biomed 2 (see 1995 External Monitoring 
Report, Figures 12 and 13), the minimum average score for the funding of proposals should be set at 
3, which is consonant with the CEC's own criteria of the borderline between 'excellent' and 'good'. 
(xii) The proportion of industrial participation, and particularly of SMEs (2.3% of contractors) is perhaps 
not as high as might have been wished or expected. The precise reasons for this are not clear, but one 
contributing factor may have been the duration of the time taken for the completion of the selection and 
funding procedure which ranged from a median of 12 to 16 months. Furthermore, SMEs may be 
concerned by the fact that although "confidential", proposals are nonetheless exposed to extensive 
peer review and are not assured of success in the competition. Nonetheless, all of the 16 companies 
who were partners of respondents to our questionnaire scored the project as important for the business 
strategy of their companies, and 9 out of 16 (56%) scored the project to be 'very important' (top 
possible score) for their business strategy. The CEC should take this as a commendation from Industry 
and as an encouragement to explore ways of fostering greater industrial participation. 
We also note that the problem of relatively few SME participants may be partly offset by the fact that 
in parallel with the main Biomed Programme, the Commission, in 1995, initiated a scheme for 
supporting Exploratory Awards for SMEs as well as Co-operative Research Projects for consortia of 
4 SMEs. The impact of this SME Initiative for the Biomed Programme will need to be evaluated at a 
later date. However, we recommend that the CEC ensures that the advantages for SMEs in 
participating in the mainstream Biomed Programme should be widely advertised (i.e. given greater 
visibility). 
(xiii) The Programme's work on illicit drugs is not as visible as the EU might have wished. This may be 
due to the fact that 'Illicit Drugs" is not a research topic in its own right, but rather a component of 
several different Research Areas. However, there are significant positive scientific advantages derived 
from the integration of 'illicit drugs' in 'Pharmaceuticals', 'Brain research' and 'Research on AIDS', 
for example. Given the high level of oversubscription of the Programme and the relatively small 
budget for Biomedicine, establishment of 'Illicit Drugs' as a separate Research Topic might be 
deleterious to the Programme as a whole unless it was matched by an appropriate increase in budget. 
96 
9. 
We recommend that the present system is retained unless special, additional sums are assigned for 
specific work on illicit drugs. 
(xiv) We are concerned by the relatively long delay between the receipt of proposals and the transfer of 
funds for research, especially since this may be one factor which might deter Industry from 
participating in the Programme. However, we are sensitive to the fact that this delay is due mainly to 
the time taken by the negotiation phase which given the small number of CEC Staff and the 
complexity of the negotiations is of necessity quite long. This delay is likely to be reduced 
significantly if the number of CEC staff was increased from 18 to 28, the number originally allocated 
to the Programme (see below). 
(xv) There is a deficiency in affirmative action in the Programme in that few female scientists are 
appointed as experts for the selection process and in the external monitoring teams. We recommend 
that the CEC takes positive steps to rectify this situation and makes every effort to become a successful 
affirmative action employer. 
(xvi) We have all participated in evaluation and monitoring processes and thereby witnessed at first hand 
the excessive workload of the CEC Staff involved in the Biomedicine Programme. The CEC Staff, as 
well as having to deal with Programme Calls, evaluations and external monitoring Committees, also 
have to manage effectively and efficiently 700, and by the end of 1997, about 1200 contracts. The 
contracts involve complex scientific and financial issues in several to many dispirit Member States. 
We believe that the excessive workload, which is due in part to the relatively small number of Staff, is 
not consistent with either good and humane work practice or efficient and effective Programme 
Management. We therefore strongly recommend that because of its central importance to European 
Biomedical Science, Health Care and Wealth Creation and because of its obvious popularity, the 
Biomedicine Programme should be awarded the 28 staff which it was originally allocated. 
(xvii) Notwithstanding some of the concerns mentioned above, we highly commend the EU for 
establishing the Biomedicine Programme which has already had a major impact on European 
Biomedical Research, and will make significant contributions to the prevention, diagnosis, 
management and treatment of diseases in Europe. Furthermore, although the proportion of Industrial 
Partners may not be as large as originally hoped for, the obvious importance of the Programme for 
industrial contractors suggests that with further targeting of the private sector, the Biomedicine 
Programme could also play a significant role in wealth creation. 
Our independent analysis has shown that the Biomedicine Programme has a high European 
Dimension and Added Value and, that the output of the Programme in terms of new European 
initiatives, health care developments, potential wealth creation and quality of publications is generally 
excellent. The Programme provides the EU with an incredibly powerful network and database of 
unrivalled expertise in Medicine from molecules through to man and Society, which in addition to 
fostering highly productive and integrated research between the Members States, serves as a vehicle 
for the rapid deployment of European Task forces to deal with strategic, and/or national issues or 
crises as they arise. The Programme can already justly and proudly claim major successes such as 
the discovery of a new variant of Creutzfeldt-Jakob Disease, the improvement of the survival rate to_ 
70% of a previously fatal disorder (severe combined immunodeficiency) and many others which are 
exemplified in our Report. The high standard, efficiency and efficacy of the Programme depends in 
large measure on the work of the CEC staff who deserve special commendation. We have identified 
several areas in which technical improvements can still be made, but the Programme provides the EU 
with a model for other research initiatives pivotal for the growth and development of Europe. 
The Biomedicine Programme offers an important mechanism by means of which individual 
biomedical talents in Member States can be harnessed for the greater good of the European Union. 
We very much hope that the Biomedicine Programmes of the Third and Fourth Framework 
Programmes will form the platform for a similar programme in the Fifth Framework, and that such a 
97 
Programme is allocated a significantly larger budget in order to reduce the wastage of many excellent 
initiatives which reflect the Biomedical power of the Member States. 
98 
IT. COMMISSION'S RESPONSE 
The Commission notes the Panel's comment that both 3rd and 4th Framework Programme activities under 
the BIOMED Programme have achieved the objectives to contribute to the improvement of medical and 
health research in Europe and satisfied the criterias of European added value, European dimension and 
subsidiarity. The Commission notes the clear distinction the evaluators observe between the BIOMED funded 
and nationally funded research which is in strict compliance with the principle of subsidiarity. Regarding the 
output of the projects supported by the Programme, the Commission notes the Panel's observation that it 
appears to be of a quality and quantity equal to or greater than those of many projects supported by National 
Bodies. 
The trend towards more high quality Shared Cost Actions being proposed and selected for funding is 
recognized by the Commission. This is taken as an indication of the increasing demand of actual research 
funding on the European level. Although the overall success rate of the programme, in terms of number of 
projects, is similar to NIH (USA) and other funding bodies, the Commission shares the concern of the Panel 
regarding the oversubscription. 
Regarding the recommendation on the allocation of funds to the various research areas, the Commission 
notes that indicative budgets for the Programme established by the Council can be and have been amended, 
depending upon research priorities, as a specific response to a call and following a positive opinion of the 
Programme Committee. The Commission also notes that a high priority is set by the Panel for research on 
illicit drugs and that this area is seen as an integral part of the biomedical research activities. This is indeed in 
compliance with the recent Council positions on the matter. 
The Commission is pleased to receive the comment on the improvements regarding the selection procedures, 
which are now described as efficient, effective and confidential, and it will continue to improve evaluation 
procedures. The recomendation that a minimum score should be established below which no proposal can be 
funded has already been implemented in the second call. The analysis of the second call shows marked 
increase in the industrial participation however, health care funding agencies (undertakings, hospitals) are a 
major participating force in the BIOMED programme. 
In terms of positive action towards having more female scientists appointed to the selection procedures, it 
must be noted that BIOMED staff is 50% male and 50% female. The Commission has only limited choice in 
the academic scientific evaluators. The ratio of male to female in the names proposed by Member States 
is 9:1. 
The Project Review Board (PRB) is the main instrument relating to the ongoing monitoring of the scientific 
progress of the Programme, and the Panel's recommendation to enhance the role of this body is noted. 
The small number of scientific staff involved in the Programme administration at present, recognized by the 
Panel, is a very serious problem and efforts are being made to find well-qualified staff in the future. 
99 
III. EVALUATION PANEL 
Prof. S. AYMÉ 
Director of Research at INSERM. A national expert in Human Genetics, Epidemiology and Public Health 
issues for genetic diseases (INSERM and CNRS) and on on rare diseases (Ministry of Health). At the 
European level, an expert for the Human Genome Project. President of the European Society of Human 
Genetics. In US, Advisory board member of the Genome Database. Member of several editorial boards. 
Member of the EUROCAT association. 
Prof. U. FAUST 
Former Head of the Instutute of Biomedical Technology in Stuttgart and long term member of the 
BIOMED Project Rewiev Board. Wide specialities on the biomedical engineering research, specially on 
medical imaging, image processing and sensor technology. Several advisory tasks for industrial R&D 
projects. 
Prof. G. FINK (Chairman and Rapporteur) 
Director of the Medical Research Council's Brain Metabolism Unit, elected Honorary Professor in the 
University of Edinburgh in 1984. Professor Fink is responsible for setting the scientific strategy of the 
Unit, deployment of the Unit's Staff and management of the Unit's core budget. His personal expertise is 
in neuroendocrinology, neuropharmacology and neuroendocrine molecular biology. His opinion is widely 
sought as a referee in the area of Neuroscience on both sides of the Atlantic. He served on the Wellcome 
Trust Panel for Mental Health and Neuroscience between 1984-1989. His refereeing has involved the 
evaluation of several major institutions. He is on the Editorial Board of several major journals. President 
of the European Neuroendocrine Association (1991-1995). Member of Council of the European 
Neuroscience Association. 
Prof. A. GRAVANIS 
Regular reviewer of research proposals funded from the Greek General Secretariat of Research and 
Technology and the Board of Biomedical Research, Greek Ministry of Health. Recognized expert on 
molecular pharmacology of hormone-dependent tumors. Autocrine/paracrine interactions of neuropeptides 
and steroid hormones in the tumors of the Repro-ductive System. Molecular mechanisms of Oncogene 
regulation in cell proliferation/differentiation. Pharmacological evaluation of new hormone antagonists in 
the hormonal therapy of cancer. 
Prof. J.M. MARTIN-MORENO 
He is a Visiting Fellow of the Harvard School of Public Health, an Associate Researcher of the Milan-
based European Institute of Oncology and a Professor in the department of Epidemiology and Biostatistics 
at the National School of Public Heath in Madrid, Spain, where he is now the Director. 
His experience ranges from implementation and evaluation of research projects at a national and 
international level to previous chairmanship of the Discipline Oriented Group on Epidemiology and 
Clinical Trials within the BIOMED I programme, and current membership of the Project Review Board of 
the European Commission's BIOMED programme. He is interested in epidemiological methods, meta­
analysis and, particularly, in the nutritional epidemiology of cancer, a subject on which he has published 
numerous papers at home and abroad. 
100 
Prof. L. NOTARANGELO 
Head, Laboratory Unit of Molecular Immunology, Department of Pediatrics, University of Brescia. Senior 
Clinical Investigator, Hematology-Immunology and Bone Marrow Transplantation Unit, Dept of 
Pediatrics, Univeristy of Brescia. Permanent Member, WHO Advisory Committee on 
Primary Immunodeficiencies.Member of the Editorial Board of several Italian and International Pediatric 
Journals. Member of the Evaluation Committee, Italian National Health Institute Research Grants (1992). 
Member of the External Monitoring Report Committee, Biomed 2 Programme (1995). 
His recent contributions include: characterization of the molecular basis of X-linked 
immunodeficiency with hyper-IgM, identification of the gene responsible for autosomal recessive Severe 
Combined Immune Deficiency,characterization of the molecular basis of X-linked thrombocytopenia, first 
European clinical gene therapy trial for adenosine deaminase deficiency. Scientific Secretary, European 
Society for Immune Deficiencies. 
101 

10. 
1996 FIVE-YEAR ASSESSMENT 
AGRICULTURE AND FISHERIES, 
INCLUDING AGRO-INDUSTRY 
103 

10. 
ï. EXECUTIVE SUMMARY 
The panel was asked to assess research programmes in agriculture, fisheries, forestry and agro-industry 
for the five years 1991-96. This period covered all of the Third, and parts of the Second and Fourth 
Framework Programmes. These included over 850 projects, with approximatel 675 MECU of EU 
funding. 
The changing balance of the programme shows a well reasoned evolution. There has been an increasing 
emphasis on forestry and fisheries, as well as on food, nutrition and environment. Most of this change has 
been policy driven. The panel felt, however, that the research programme should lead policy, as well as 
follow it. This means that there should be a more explicit place in the programme for the pursuit of longer 
term goals. 
The panel undertook a direct assessment of a randomly selected sample of 15% of the projects in AIR. 
Projects were scored for progress in relation to scientific goals, potential impact, effectiveness of 
dissemination, and linkages created. For all four criteria, at least 82% of projects were judged to have 
performed "in line with expectation" or better. Based on the detailed results of this assessment, and the 
evidence from existing reports on other programmes, the panel concluded that the programme as a whole 
is very successfully meeting its objectives. 
The replacement of the separate sectoral programmes of FP2 with the integrated AIR programme of FP3, 
and FAIR in FP4 has worked well. Co-ordination between the three DGs concerned appears good. 
Linkages to eight parallel programmes are important, but there is little information on how effective they 
are. 
Despite some excellent examples (e.g. AIR project catalogue) the overall information system on EU 
programmes is not well adapted to facilitate analysis and management. 
The information base from which to analyse and co-ordinate linkages between the EU programmes and 
national programmes is even less satisfactory. 
Broadly speaking, the programmes were very well managed. This judgement was supported by the 
opinions of 378 project co-ordinators who responded to a survey. The panel has made a number of 
suggestions (see Recommendations) for improvement in the management procedures. 
One area requiring additional attention is dissemination. Communication of results to the scientific 
community appears to be excellent, but communication to intermediate and end-users could be improved. 
The FLAIR FLOW model was judged to be a good example. 
Recommendations : 
1. The research agenda should provide for more work on longer term issues, beyond the immediate policy 
objectives, and on systems and themes which may not be economic today. 
2. Research across the spectrum of activities encompassed in the term "sustainability" should be more 
strongly encouraged. This should be done by delineation of areas where increased work is seen to be 
necessary, as well as by including a sustainability criterion in the guidelines on projects. 
3. There should be increased emphasis on research which responds to the growing consumer demand for 
information on and confidence in the integrity of the food chain. 
4. The agenda should encourage a proportion of projects which reach out to the leading edge of scientific 
development. 
5. Linkages to the environmental program need to be increased. 
6. Linkages to national programmes need to be improved, initially by developing an effective common 
database. 
105 
10. 
7. The present procedure for project submission could be improved by providing for submission of outline 
proposals in a pre-screening phase. 
8. On completion, each project should be subject to an independent review. 
9. More effective dissemination of results is needed. This applies particularly beyond the level of 
scientific publication, which seems adequate. The FLAIR FLOW example could be a model in some 
areas. Better promotion of existing mechanisms (Mobility, Dissemination) in connection with the 
research programme might also be useful. 
10. Greater involvement of industrial partners, particularly SMEs, should be encouraged. 
11. Concerted Action type activities should be increased. 
12. The management of the programme by the Commission Services needs to be strengthened, in some 
areas by the deployment of additional staff, and particularly by the investment of more resources in 
an efficient Management Information System. 
106 
II. COMMISSION'S RESPONSE 
The Commission welcomes the external evaluation report concerning Community research programmes in 
the field of agriculture, fisheries and agro-industry for the five-year period 1991-1996. The report 
provides a good insight of the evolution of successive programmes and the Commission is pleased with the 
overall positive opinion. The recommendations made by the panel will be considered when implementing 
the remaining part of the Fourth Framework Programme and when preparing the Fifth Framework 
Programme. 
The objectives of Community research programmes in the fields of agriculture, fisheries and agro-industry 
have been manyfold but the most crucial aims include: to promote and harmonize trans-European research 
and build a strong scientific basis in Europe; to strengthen the competitiveness of European industries in 
these sectors, and to provide support to Community policies. The latter objective does not necessarily 
imply immediate needs, for which other sources of funding often exist, but mostly involve long-term 
policy objectives (e.g. rural development, sustainable exploitation of fisheries resources). Similarly, many 
basic research areas such as plant and animal health, generic science for nutritious foods or the biology of 
new aquaculture species have been included in previous and current EU programmes. Furthermore, many 
cross-links have been established with other specific EU programmes involving some more basic research 
aspects related to the Biotechnology, Biomedicine, Energy, Materials and Industrial Technologies, 
Environment or Training and Mobility programmes. In all areas of the previous workprogrammes, 
scientific and technological innovation stands as the most important criterion for selection and it is strongly 
underlined in calls for tender. 
Sustainability issues have received increasing attention during the five-year period and are currently 
addressed in many areas of the FAIR workprogramme. The sustainability criteria suggested in the report 
will be investigated. Linkages to the environmental programme currently exist through interservice 
consultations and meetings for specific areas of the workprogramme. In the context of the Fifth 
Framework Programme, research in the fields of life sciences and environmental sciences are likely to be 
more closely related. 
Several attempts have been made to overcome the difficulties and limitations associated with linkages to 
national programmes. A directory of fisheries and aquaculture research organizations including their 
research activities has been successfully produced and an inventory of agricultural research projects has 
been supported in an initiative named AGREP which is an information system for ongoing, publicly 
funded research projects currently implemented in 11 EU Member States. Further to the Commission's 
communication 'Achieving coordination through cooperation', a CREST ad hoc group has been 
established in 1996 for facilitating the exchange of information on national RTD activities. During the 
first year of activity, the group has compiled a large amount of relevant national information and has 
produced a valuable report. Concertation activities have continuously been strongly encouraged and 
priority has always been given to funding of concerted action proposals at the selection stage. The concept 
of thematic network has been promoted in the FAIR programme to reinforce concertation activities. 
Efforts will be pursued to stimulate high quality concerted actions, particularly in areas of strategic 
importance for Europe. 
The level of industrial participation in projects strongly depends on the research areas involved. Agro-
industrial projects involve almost always at least one industrial partner and show an increasing industrial 
participation during the five-year period with a current involvement higher than 25 %. In primary 
production areas, industries are usually not technology oriented and therefore have not been very active in 
European research programmes. The technology stimulation measures for SMEs which have been put in 
place in FAIR have attracted a growing number of proposals and have proved to be an efficient tool for 
raising SME participation particularly in the food technology domain. 
107 
10. 
Dissemination of results is a vital component of the Community research programmes. While 
dissemination to the scientific community has proved to be very effective through scientific journals and 
conferences, dissemination to end-users requires more careful attention. Specific networks have been 
established in the food and non-food sectors, whereas dissemination in other domains of the programme 
relies more on national initiatives and dissemination plans implemented in each project. Dissemination of 
results to SMEs will be ensured through the recently established FAIR network of focal points. Consumer 
trends in general and the growing consumer demand for information on and confidence in the integrity of 
the food chain in particular have been increasingly addressed in successive programmes and are currently 
specifically identified in the context of accompanying measures on ethical, legal and social aspects. Indeed 
the role which food plays in human health has been given increasing importance starting from a small base 
in FLAIR, continuing in AIR and representing now a significant area in FAIR. 
Programme management improvements are continuously sought. While informal pre-screening has been 
very active for guiding applicants to the most appropriate sources of funding, it is felt that a formal pre-
screening step could create even bigger disappointments among proposers and that targeting calls for 
specific areas could be a more effective mechanism to combat oversubscription. Various ways for 
strengthening project monitoring and assessment after completion are also being considered including the 
assistance of external experts. Finally the need for an efficient information management system is 
recognized by the services. New information systems have been recently developed (e.g. AMPERE) but 
further improvement are necessary to allow harmonized and up-to-date information management. This 
issue could also be resolved by increasing human resources at the Commission level. 
In conclusion, most of the recommendations made by the panel prove to be consistent with the first 
orientations for the Fifth Framework Programme, proposed by the Commission in 'Inventing tomorrow'. 
The Commission strongly believes that this independent evaluation of EU RTD activities will thus be very 
valuable for orienting further implementation of the Fourth Framework Programme and for preparing the 
Fifth Framework Programme. 
108 
m. EVALUATION PANEL 
Prof. Stefano Cataudella, Università degli Studi di Tor Vergara (IT) 
Professor Cataudella, Associated Professor in the University Tor Vergara (Rome) has a wide experience in 
research, especially in aquaculture-environment relationships and in genetic characterization of reared 
population of fish. Concerning academic experience he has been involved in different university 
departments since 1973 and co-operated with international programmes for development. He also 
participated in the evaluation of FAR projects. 
Prof. Patrick Cunningham, Trinity College Dublin (IE), Panel Chairman 
Professor Cunningham was Director of the Animal Production and Health Division at the FAO until 1993. 
Since then, he returned to his position as Professor of Animal Genetics at Trinity College and is very 
active in various international advisory committees. His research work has been centred on quantitative 
genetic theory and on the efficiency of livestock improvement programmes. 
Mr Francisco Javier Pereiro Mufioz, Instituto Espanol de Oceanografia (ES) 
Mr Pereiro Mufioz has a twenty-year experience in fisheries research , especially in demersal species in 
European waters. He is the head of the IEO Fisheries Research Programmes in European waters and has 
been member of the Scientific and Technical Committee for Fisheries (STCF) of the EC. He is Spanish 
delegate in some committees of the International Council for the Exploration of the Sea (ICES) and in the 
ACFM (Advisory Committee for Fisheries Management). 
Dr Daniel Richard-Molard, Institut National de la Recherche Agronomique (FR) 
Dr Richard-Molard is currently the Head of the Protein and Carbohydrate Technology department as well 
as the Fruit and Vegetable Biotechnology department at INRA. His research work is focused on the 
transformation technologies of cereals for food and non-food purposes and involves many cooperation 
programmes with industry. 
Prof. Jorma Sundquist, The Finnish Pulp and Paper Research Institute (FI) 
Professor Sundquist was the Research Director of the Paper Science Centre at the Finnish Pulp and 
Research Institute. His research interests cover polymer science, wood and pulping chemistry as well as 
chemical textile technology. He also participated in the evaluation of FAIR projects. 
Prof. S. Athanasios Tsaftaris, Agricultural University of Thessaloniki (GR) 
Professor Tsaftaris's research interests include many different aspects of plants, particularly biochemistry, 
physiology, genetics, breeding and biotechnology. Professor Tsaftaris is currently serving in the National 
Research Council of Greece and is a member of several international scientific advisory boards. 
Prof. Marcel Vanbelle, Université Catholique de Louvain (BE) 
Professor Vanbelle's area of expertise covers animal and human nutrition. He has been a member of the 
Belgian Council of Human Nutrition since 1990, has published more than 450 papers and is a member of 
several international scientific advisory boards. 
Prof. Heinrich Wohlmeyer, Agricultural University of Vienna (AT) 
Professor Wohlmeyer is an agricultural scientist with international training in economics and commodity 
agreement. An industrial experience in agro-industry preceded his current research position at the 
University. His research interests involve natural resource management and environmental economics. 
109 

Il 
1996 FIVE-YEAR ASSESSMENT 
NON NUCLEAR ENERGY 
11! 

11. 
I. EXECUTIVE SUMMARY 
1.1 Background 
This is a self-contained summary of the report of the Assessment Panel on the Non-Nuclear Energy (NNE) 
RTD&D activities of the EC for the period 1991-1996. The summary forms an integral part of the main 
report of the Panel. The Assessment was carried out by a panel of independent experts, whose names and 
Terms of Reference are annexed to the main report. These RTD&D activities took place under the JOULE 
Programme within the Third Framework Programme and under the THERMIE Programme, and under 
their successor, the JOULE-THERMIE Programme, within the Fourth Framework Programme. 
As no final evaluation of these programmes was carried out under the Third Framework Programme, this 
Assessment subsumes the evaluation of the activities under the Third Framework Programme, in 
accordance with the Commission's Communication to Council COM(96) 220 (final) of 22 May, 1996. 
1.2 The NNE RTD&D activities 
The oil crises of the seventies spurred the EC interest in energy research and demonstration, with the 
research element being managed by DG XII and the demonstration element by DG XVII. The energy 
research programme became known as the "JOULE" Programme in 1989 under the Second Framework 
Programme, and the demonstration element as the "THERMIE" Programme, which was not part of the 
Framework Programme. The two elements were amalgamated to form the "JOULE-THERMIE" 
Programme in 1994, under the Fourth Framework Programme. The two elements continue to be formally 
administered and managed by DG XII and DG XVII, respectively, and to have two separate management 
committees. 
The objectives of the two programmes and their successor are described in Section 3.1 and in Annexe 2 of 
the main report. As a generalisation, project co-ordinators in the JOULE Programme tend to be located in 
universities and research organisations, whereas those in the THERMIE Programme are industrial 
enterprises or industrial research institutions. While individual project budgets under THERMIE tend to be 
larger than those under JOULE, the overall programme budgets are comparable. 
The technical activities within the NNE programmes cover a wide domain including Renewables Energies, 
the Rational Use of Energy, and Fossil Fuels. In addition to research and demonstration actions, the 
domain includes associated measures such as supporting and dissemination actions, and strategic studies. 
1.3 Structure of the main report 
The main report has been structured in accordance with the format suggested by CREST for the 
assessment of all specific RTD&D programmes. 
- Section 1 presents a self-contained summary of the main report. 
- Section 2 is an introduction to the report and to the methodology used. 
- Section 3 provides a detailed critique of the programmes. 
- The Assessment Panel's conclusions are presented in Section 4. 
- The Panel's recommendations are presented in Section 5. 
- Supplementary information is contained in the Annexes. 
1.4 Methodology of the Assessment 
This assessment was carried out at the programme level, not at the project level. A questionnaire was sent 
by the JOULE and THERMIE managements to a random sample of 340 project co-ordinators. The 
questionnaires were returned directly to the rapporteur over a period of months. Analyses of these 
113 
11. 
questionnaires are presented in Annexes 3 and 4. Background information on the programmes was 
provided by the Commission services to this Panel at the beginning of, and during, the Assessment. The 
Assessment panel designed a questionnaire, which was distributed, with the permission and co-operation 
of the Programme Directors, to programme managers and staff responsible for the JOULE and 
THERMIE elements of the Programme. Responses were returned anonymously to the rapporteur, and are 
presented in summary form in Annexe 5. The Assessment Panel interviewed the Directors of the JOULE 
and THERMIE programmes with some of their staff. 
The Assessment Panel first met in Brussels on 2nd September, 1996 and on five occasions between then 
and 20th December, 1996. 
1.5 Assessment of the NNE RTD&D activities 
By far the greater part of the EU's energy supply still comes from non-nuclear sources and will continue 
to do so for the foreseeable future. The programme's concern with non-nuclear energy is therefore still 
justified. There remain concerns about security of energy supply, environmental impacts and other issues. 
The objectives of the Community's actions in the domain of NNE RTD&D were to: 
- Increase the long-term security of energy supply 
- Reduce consumption and imports of energy 
- Alleviate environmental problems related to energy production and consumption. 
These objectives were to be served through carrying out RTD&D on the rational use of energy and 
exploitation of renewable and new indigenous sources of energy. They were reinforced and broadened to 
take account of Union policies and of socio-economic and political changes. The initial objectives of the 
programmes are still valid against evolving scientific, technical, industrial and socio-economic conditions. 
In the Assessment Panel's opinion they will remain largely valid for the Fifth Framework Programme, 
taking account of the availability of energy technologies and of their slow market penetration. 
Selection criteria, procedures, objectives and scope of the activities as laid down in Council Directives 
have been respected by the Commission. However, it has proved difficult to adopt specific and verifiable 
Programme objectives against which performance can be evaluated. This is a common problem in 
research. 
In the Panel's opinion, the NNE activities have been managed in a cost-effective manner. The staff 
resources allocated to management are, if anything, somewhat inadequate resulting in overload and delays. 
Nonetheless, the management of the activities is generally satisfactory and efficient. Information provided 
to proposers has improved in quality in recent years and the project selection process in the JOULE 
element is transparent. Although the NNE activities fall within one programme, the JOULE-THERMIE 
Programme, there are in fact two management committees and there are legal separations between the two 
elements. An external monitoring process was introduced in 1995 and has proved useful to this Panel and 
to the Commission services. 
The management of the Programme has appropriate modes of action open to it and adequate flexibility in 
operation. However, this freedom has not always been utilised to best advantage. 
The Assessment Panel found it difficult to judge the effectiveness of the Programme or the achievement of 
objectives. This is a problem common to all research activities which have broadly-stated objectives, and 
is compounded by the fact that national governments and private organisations also carry out NNE 
RTD&D, making the Commission's achievements difficult to isolate. However, there are areas in all three 
main sectors of the Programme (Rational Use of Energy, Renewable Energies and Fossil Fuels) where the 
NNE activities have given the Union a leadership position which national efforts would not have done as 
well. Particular examples of successful intervention are to be found in fuel cell technology, wind turbine 
design and combustion in automotive engines. The Community can play a particularly effective role in 
114 
11. 
supporting long term and risky transnational research projects which individual Member States might have 
difficulty in supporting. 
Replication and commercialisation have not always been as successful as would be desired, due in part to 
the low price of fuels and to the apparent security of supply. The response from contractors to the 
questionnaires, although disappointing in some respects, did indicate substantial outputs in the forms of 
publications, theses, new products, new processes and test methods. 
The scientific and technological base of the Union has undoubtedly been strengthened by the NNE 
activities through the development of research teams and research facilities, and the promotion of 
international co-operation. In addition, the NNE activities are consistent with the Community's energy 
policy and have supported other Community policies on the environment, cohesion, the development of 
the internal market and sustainable development. There are in fact at least twelve Community initiatives 
and instruments related to the NNE activities which serve to exploit and disseminate research results. The 
linkages between these programmes and instruments are still being developed. Coordination between the 
Community's activities and those of Member States has not always been good, but is improving. 
1.6 Conclusions 
The Assessment Panel's conclusions are summarised below: 
The initial objectives of the programmes are still valid against evolving scientific, technical, industrial 
and socio-economic conditions. These objectives will remain largely valid for the Fifth Framework 
Programme. 
Community-funded RTD&D has contributed to almost all the developments in the field of NNE and 
has also helped strengthen the Union's scientifc and technical base. Funding has been provided for 
long term and risky research, which might not have been funded otherwise. 
De facto, the two elements of the Programme operate largely as two separate programmes, as they 
originated in two different Directorate General. 
The management of this programme is considered by this Assessment Panel to be cost effective. The 
transparency of management is generally satisfactory. 
- Both the THERMIE and JOULE elements of the JOULE-THERMIE Programme appear to be under­
staffed. This has resulted in delays in contract negotiation and insufficient time for real project 
management. 
Delays in contract negotiation continue to be a problem for contractors and appear to be caused in 
part by delays in the contracts division. 
Clear quantified objectives have not been set for the Programme, making if difficult to assess its 
effectiveness. It might be difficult to set such objectives. 
Co-ordination of Community and national programmes is not satisfactory. 
Full use does not appear to have been made of all available modes of action. 
The project selection criteria and procedures, as well as the objectives and scope, laid down in 
Council directives have been respected by the Commission. 
Literature provided for intending participants in the Programme has improved, but there is room for 
further improvement. There is also scope for harmonising the procedures for project selection in 
JOULE and in THERMIE. 
It is difficult to fund good projects which contain elements of research as well as demonstration. 
Project evaluations do not fully use metrics such as milestones and outputs. 
Evaluation of completed projects within the JOULE element of the Programme has hitherto been non­
existent, but is now in train. 
- The establishment of the annual monitoring exercise is of great assistance to the programme 
management. 
115 
11. 
1.7 Recommendations 
The Assessment Panel recommends: 
That RTD&D activities should continue in the area of NNE. 
The NNE Programme should be more oriented towards user needs, emphasising applications of 
technologies, but a portion of the budget should still be directed to basic research. High quality 
standards must be maintained at all times. 
Further action should be taken to harmonise the operation of the JOULE and THERMIE elements of 
this programme, for example by having a unified management structure and by having harmonised 
proposal evaluation and selection procedures. 
Better integration of THERMIE type A and type B actions should be assured to improve replication 
rates. 
Monitoring and evaluation procedures should be extended and improved to include mid-term reviews, 
clearly defined milestones and annual reporting with project highlights for both the JOULE and 
THERMIE elements. The monitoring panel should have continuity and a member of the monitoring 
panel should serve on the Assessment Panel. 
More use should be made in the JOULE element of the Programme of all of the modes of action and 
of the flexibility open to the programme management, such as concerted actions. 
Adequate resources should be devoted to programme and project management, inter alia to reduce 
delays in contract negotiation. 
Better co-ordination with, and between, Member States' programmes should be established. 
A procedure should be devised to deal with good proposals which overlap the JOULE and THERMIE 
elements of the Programme. 
116 
11. 
II. COMMISSION'S RESPONSE 
The Commission welcomes this independent evaluation of Research, Technological Development and 
Demonstration (RTD&D) activities in the field of Non-Nuclear Energy (NNE), prepared by the high level 
group of experts chaired by Dr. Eamonn Kinsella, and is grateful for the effort shown by the panel in 
assessing the last five years of the programme. 
The report comments on an important number of elements associated with the relevance of the programme 
objectives, efficiency and effectiveness of overall management and implementation mechanisms. The 
Commission views the remarks as providing helpful guidance and takes good note of the Assessment 
Panel's conclusions that initial objectives of the programme should remain largely valid for the future; that 
Community-funded RTD&D has contributed to almost all the developments in the field of NNE and has 
also helped strengthen the Union's scientific and technical base; that project selection criteria and 
procedures, as well as the objectives and scope, laid down in Council decisions have been respected; and 
that the overall management is considered to be cost effective. 
The Commission welcomes the Panel's recommendation to orient the NNE programme more towards user 
needs, emphasizing the application of technologies. In fact, the provision of high-quality energy 
technologies and services - which respond to the citizens expectations for improved quality of life - and the 
timely demonstration that these technologies and services will limit environmental hazards are essential 
elements of the European Energy policy. 
The Commission supports the Panel's view that the objectives of the Community's actions in the domain 
of NNE are served through carrying out RTD&D actions on rational use of energy, exploitation of 
renewable and new indigenous sources of energy and also in the hydrocarbon sector. 
The Commission is also aware that additional efforts are needed, at the whole framework level, for 
harmonizing management and administrative procedures as well as improving co-ordination with, and 
between, the various Member States' RTD&D programmes. The Communication "Inventing Tomorrow" 
and the subsequent Working Document on the Structure, Instruments and Objectives of the Fifth 
Framework Programme indicate the main lines of actions which are intended to be followed in this 
respect. 
Finally, the Commission views the recommendations as very constructive, gearing both future activities 
and identifying short term adjustments which are due to be timely introduced in the remaining parts of the 
current programme. 
117 
11. 
III. EVALUATION PANEL 
Dr. Eamonn KINSELLA 
Mr. E. Kinsella is Director of a consultance company The Circa Group Europe. He is an expert in energy 
and in material technologies. He was a Research manager at Moy Insulation Ltd and a Member of the 
National Board for Science & Technology (Ireland). 
Mr. Maurice CLAVERIE 
At the CNRS, Mr. Claverie is responsible for the relations with the Public Research Establishments and 
coordinator of the PIR (Programmes Interdisciplinaires de Recherche). He is also advisor at the DGRT 
(Direction Générale de la Recherche et de la Technologie) for the energy and environmental problems. 
Mr. Dale LAIDLER 
Mr. D. Laidler has joined ICI Corporate Laboratory as Senior Research Scientist. Since 1991, he is the 
External Collaborations Manager. The role involves helping individual research groups within ICI to become 
involved in academic and inter-company collaborative activities. 
Mr. Hans Christian SORENSEN 
Director at the Energi og Miljoundersogelse company, Mr. Sorensen is specialised in the field of rational use 
of energy and use of renewables in buildings and in the area of combustion. 
Prof. Anastasios KARABELAS 
Mr. Karabelas is Professor in the Chemical Enginnering Department of the Aristotle University of 
Thessaloniki. He is Director of the Laboratory of Chemical Process and Plant Design. 
Prof. Dr. Ing. Dr. Ing. E.h. Helmut SCHAEFER 
Mr. Schaeffer is Professor at the Technical University of Munchen. He is responsible for the Research Unit 
for Utilities. He has experience in applied research in the area of utilisation of energy in industry, in domestic 
and transport (structural analysis in energy demand, energy concepts, ...). 
118 
12. 
1996 FIVE-YEAR ASSESSMENT 
TRANSPORT 
119 

12. 
T. EXECUTIVE SUMMARY 
Recommendations 
1. It is very important that transport remains a distinct programme, while being strongly connected to other 
areas of the Framework Programme. 
2. The subsequent overall transport RTD programmes should include transport related aspects like 
technology development, energy conservation, social and environmental considerations, land use etc, with 
the various aspects managed by the appropriate DGs within a coherent framework. 
3. It is important to keep a mixture of technology research and policy research within one overall transport 
RTD programme. 
4. Tasks to be included in the subsequent programmes should be selected so that highest priority is given to 
problems that most effectively are handled on an European level and especially those which are of 
importance for the EU transport policy. 
5. It is important that the overall programme has a clear and communicable vision; there should be scope for 
selected risk taking and also some long range research that spans successive Framework Programmes. 
6. The specific transport programme should be structured so that there is a strategic part, an "umbrella", 
where problems common to the different modes, and co-ordination and competition between modes are 
handled. Under this "umbrella" relations between transport and other related aspects should be included. 
7. There is an urgent need to increase the number of staff to manage the transport RTD activities; initially up 
to the agreed staffing levels but a policy-oriented programme of this nature needs a higher ratio of 
scientific officers to projects than in more technology-oriented programmes, so agreed levels should be 
raised. 
8. There is a need to build in more flexibility in the programming of the budgets. A certain amount of money 
should be kept aside within the specific programme to make it possible to react to changes in society and 
policy, new developments in technology, to prolong projects from earlier programmes, etc. 
9. We strongly support the measures DGVH are proposing to take to improve dissemination; we emphasise 
the need for dedicated in-house staff with specific responsibility to facilitate dissemination and recommend 
that appropriate resources are made available. 
10.Concerted actions and task forces have proved to be very effective in enhancing co-operation and co­
ordination, and these mechanisms should be built appropriately into future Framework Programmes. 
Background 
This assessment report has been prepared according to Article 4.2. of the Council Decision of 15th 
December 1994 on the specific RTD programme in the field of Transport. The programmes and initiatives 
covered by this 5-year-assessment are: EURET, APAS and the start-up phase of the much larger Transport 
RTD Programme in FP4. 
The objectives of this five year assessment are to consider the relevance, efficiency and effectiveness of the 
programmes and initiatives and the contribution of the RTD activities to meeting the Community's transport 
and other policy objectives. 
121 
12. 
This assessment was carried out between July and October 1996. The Panel examined project and evaluation 
reports and the documentation associated with the Transport RTD Programme. In addition, the evaluators 
have held a number of meetings with project leaders, representatives from related EU projects, members of 
the Transport Research Committee (TRC) and staff from DGVII, DG XIII and Eurocontrol. The evaluation 
panel met on four occasions at the offices of DGVII, and corresponded directly at other times. 
The intention of the transport research programme is to improve the efficiency of the individual transport 
modes and to find a strategy for a faster integration and interoperability. The research covers both 
passenger and freight transport. There are other specific programmes that are related to transport, which are 
evaluated separately, but we have included some connections to these other programmes. 
Major achievements 
It is only from the limited budget EURET and APAS activities that major achievements can be expected, 
as the Transport RTD work is still in its early stages. Examples of major achievements include the 
following: 
Intermodal: design of an innovative and efficient system of rapid loading and unloading of goods between 
different modes of transport (SIMET). 
Rail: development of a harmonised European Rail Traffic Management System (ERTMS) aimed at 
overcoming the fragmentation of rail traffic management systems used by the national railways. 
Waterborne (maritime): application of advanced technologies and their integration with optimum crew 
composition, manning and operational strategies (ATOMOS). 
Air: improvement and evaluation of air traffic management scenarios (AEGIS). 
Road: use of cost-benefit and multi-criteria analysis for new road constructions. 
In addition, the research activities have been very successful in encouraging the relevant groups in 
industry, transport operators, researchers and policy makers to work more closely together, and in 
fostering a greater degree of co-operation between organisations in different Member States. 
122 
12. 
II. COMMISSION'S RESPONSE 
The Commission welcomes this independent evaluation of EU RTD in the field of transport. The 
recommendations will be taken into account in orienting the further development of transport related 
research in the 4th and 5th Framework Programmes and will also have an impact upon the way in which 
the results of the research and development are disseminated and implemented. 
The Commission acknowledges the need for a mixture of "technology" research and "policy" research. It 
welcomes the panel's conclusion that "the contribution of the specific transport RTD programmes to 
assisting in the implementation of the transport policy of the EU is considerable". This requirement to 
support policy is clearly set out in Article 130f of the Union treaty which refers to the need for research to 
strengthen the scientific and technological bases of Community Industry and to support other Community 
policies. It is also clearly acknowledged in the Communication from the Commission "Inventing 
Tomorrow".1 
The panel desires to see a single Transport RTD Programme in FP5. The need for a degree of 
concentration of future research efforts, has been recognised by the Commission. The FP5 proposal is 
likely to see a significant reduction in the number of specific programmes. This would facilitate the co­
operation and co-ordination of research effort on themes such as transport which are currently situated 
within four different specific programmes. 
The Commission also notes the opinion of the panel with regard to the need for an overall "umbrella" for 
transport RTD under which problems common to the different modes, and competition and co-ordination 
between modes, are clearly handled. The specific Transport RTD Programme has devoted around 20% of 
its budget to Strategic Research which is in part fulfilling such a role. 
As regards flexibility, the Commission recognises the need for flexible programming and management in 
the context of the Transport RTD Programme. Use has already been made of the limited flexibility 
provided for by the specific Programme on Transport. Examples include the modification of the funding 
basis for a number of tasks and the emphasis on aspects of research related to road pricing introduced in 
the Third Call for Proposals. The creation of greater flexibility, through setting aside a certain percentage 
of funds within a specific programme to cover unforseen requirements, will however be considered when 
preparing the Fifth Framework Programme. 
The role of concerted actions and task forces has rightly been recognised by the panel as a means for 
enhancing co-operation and co-ordination. The Commission intends to incorporate both of these types of 
actions in future Framework Programmes. 
Finally, the Commission recognises the importance of dissemination and exploitation in the field of 
transport RTD. It insists, within the Transport RTD Programme, that all projects include a work package 
dedicated to the dissemination and exploitation of RTD results. In addition, a specific task at a 
programme level was launched in the 3rd Call for Proposals on 17 December 1996 which aims at a range 
of measures to foster the dissemination and exploitation of all research results. 
1 COM (96) 332 final - 10.07.1996 
123 
12. 
IIL EVALUATION PANEL 
Professor Peter Jones 
Director, Transport Studies Group,University of Westminster. UK 
Professor Bengt Holmberg 
Professor, Dept. of Traffic Planning and Engineering, 
University of Lund SW 
Mr Ambrosius Baanders 
Deputy Director, 
Transport Research Centre AW NL 
Mr Jorge Hernando Cogollor 
President, Consultrans S.A. ES 
Mr Harri Kallberg 
Director, Road Traffic Centre. SF 
Dipl-Ing Konrad Lukaschek 
Head of Division, Signalling and Traffic Control Systems 
SIEMENS AG Ôsterreich. AU 
124 
13. 
1996 FIVE-YEAR ASSESSMENT 
TARGETED SOCIO-ECONOMIC RESEARCH 
125 

13. 
T. EXECUTIVE SUMMARY 
1.1 The Panel's general findings on the Targeted Socio-economic Research Programme 
Before the start of the 4th Framework Programme, socio-economic research at Community level was 
characterised by two processes: On the one hand concrete social science research and studies were 
performed in connection with the MONITOR programme and as an integral part of some specific 
programmes. On the other hand, a broad discussion on the potential importance of collaborative European 
social science research started in the late 1980s. Major institutional actors contributed substantially to that 
discourse. 
The 4th Framework Programme followed the ambitious plans that were the outcome of that intensive 
interactive process only to a very limited extent. The scope of the TSER programme is not only much 
narrower than the different considerations in the beginning of the nineties have envisaged, but it is also 
divergent from them in so far as it is highly fragmented in nature. Thus, it appears more as a first pilot 
activity in the area of socio-economic research at European level and less as a strategic tool to investigate 
into the deep roots which give way to exclusion, poverty, unemployment, urban deterioration etc. in 
Europe. 
The assessment Panel has concentrated on the TSER programme as it is, not on TSER as it could or 
should have been. In addition, it must be emphasised that the Panel's assessment is based on insights and 
impressions obtained at a very early stage of the programme's development, when projects have started 
only a few months ago. Keeping these points in mind: 
a. despite its limited scope, the TSER programme must be considered as a first important attempt to 
improve the knowledge on a number of important specific problems and to encourage large scale co­
operation in the European social science community thus providing inputs into the political debate and 
contributing to the development of a European socio-economic research infrastructure; 
b. although the programme has seen many different challenges, mainly on the general leadership and 
management level in addition to the typical learning problems of a fairly new activity, a number of high 
quality research projects and thematic networks have been selected and started to work a few months 
ago. 
The TSER programme is providing a new platform for collaborative multi-national, multi-cultural, policy-
oriented and policy-relevant research, which is a new feature for socio-economic research in Europe. The 
programme objectives are more valid than ever. The Panel expects that relevant results will be produced, 
in parts of the programme already in the very near future. The Panel was pleased that in some of the 
projects some interesting results are available already. The results of the programme will contribute to the 
development of new scientific knowledge in the areas of socio-economic research covered by the 
programme, and to the improvement of Community policies in the areas of research and technological 
development, education and training, and social exclusion and integration. 
Since the programme's specific objectives call for extensive networking as a special characteristic of 
TSER, a precondition for safeguarding the success will be that the research/policy links are further 
developed - at the level of the programme, the thematic clusters and the individual projects. First results 
are promising, but appropriate mechanisms of intensive interaction and communication have to be ensured 
and strengthened also in the future. 
The TSER programme has been facing many organisational difficulties in its development - already four 
different directors in its short life time, changes in personnel and responsibilities, and a number of 
different organisational structures. Obviously, the Commission was not able to ensure a harmonious start 
up phase of that new activity until recently. For the Programme Committee the leadership problems at the 
programme level in the first period of the programme development have been less then satisfactory. 
127 
13. 
However, the Panel acknowledges that during the implementation phase of the programme the 
Commission staff performed surprisingly well under very complex, turbulent and sometimes confusing 
circumstances, and succeeded to organise the start up of the programme - call for proposals, project 
evaluation and selection, contract negotiation - efficiently and to a high standard. The Panel is satisfied to 
note that for the second Call for Proposals the recommendations of the 1995 External Monitoring Report 
have been followed. 
The objectives of the TSER programme as defined in the Council Decision are defined sufficiently clear. 
In the future, however, target groups and performance indicators should be defined where appropriate. 
The present Panel is aware of the criticism by the previous 1995 External Monitoring Report of the TSER 
Programme that the management costs of the TSER Programme are too high. After one year of experience 
the Panel takes a different perspective on this issue. The specific nature of the programme in connection 
with its important networking tasks calls for adequate staffing - both in numbers and in appropriate 
qualifications. In the very near future, the situation will become even worse, when the projects resulting 
from the second Call for Proposals will be starting. 
The TSER programme has appropriate modes of action - research projects, thematic networks, concerted 
actions and accompanying measures. The Panel would welcome that thematic networks and concerted 
actions are stronger advertised, because these modes of participation in the programme are especially 
suited to mobilise and stimulate European co-operation in the socio-economic research community where 
this was not developed so far in the past. In addition, the Panel strongly support the attempts to cluster 
projects around themes following similar objectives, thus providing a mechanism for re-integration of an 
originally fragmented programme. 
The quality of the selected projects leads the Panel to believe that they will generate considerable added-
value. Thus, the Panel expects a considerable impact of the programme - in economic terms through its 
future contribution to the improvement of Community RTD policies, and in terms of inputs to the 
development of new European approaches in education and training as well as of new social models. Also 
its potential contribution to harmonisation of social data has to be emphasised. The TSER Programme has 
initiated a process of Europeanisation in the different areas of socio-economic research addressed so far. 
This can be seen as an important impact in itself already, because a well developed European social 
science infrastructure will be needed in the future. 
However, as has been said already, the achievement of the objectives will to a large extent depend on the 
effectiveness and efficiency of the research/policy interaction. 
The Panel has identified deficits with regard to the communication, networking and co-ordination tasks of 
the TSER programme, in the direction of other specific programmes, other Commission services and of 
the policy levels in the Community and in the member states. These deficiencies are clearly linked to the 
staffing problem of the programme. 
The Panel strongly supports a dual approach to the integration of social sciences in the 5th Framework 
Programme: 
. The Panel firmly recommends a separate specific socio-economic research programme addressing a 
well chosen, coherent set of themes that are European by their very nature. A specific programme will 
provide the visibility and the integrative momentum that will be necessary to intensify the formation of 
a European social science area, to support other Community policies and to underpin the activities of 
other Community programmes and Commission services. 
. The Commission should continue and strengthen the social sciences parts of specific programmes where 
appropriate. It will be necessary to further develop specific methodologies for risk and impact 
assessment in the different technology areas, so as to arrive at integrated planning systems that will help 
avoiding negative social, environmental and economic impacts from the outset. 
128 
13. 
The credibility of the European Union as a supra-national organisation will depend to a large extent on its 
success in solving the most challenging problems in the different policy areas connected with the overall 
process of European integration. The Panel is convinced that there is a need for integrated, system-
oriented approaches as provided by a well designed future socio-economic programme. 
1.2 The Panel's major recommendations 
In the following the Panel's major policy relevant recommendations are given in a condensed form. A 
much more detailed list is given at the end of the report. 
1.2.1 Contents of the TSER programme 
1.2.1.1 Programme design, workprogramme etc 
1. The Panel supports the notion of European multi-cultural, multi-disciplinary, policy-oriented and 
policy-relevant socio-economic research organised in a separate specific programme for socio­
economic research. 
2. The Panel recommends to discuss in the Programme Committee a revision of the work programme 
for the 3rd Call for Proposals, restricting the contents to a few strategic themes and ensuring 
synergies between the different areas of the programme. The new work programme should indicate 
well defined target groups, clearly defined objectives, as well as research tasks and performance 
indicators. 
3. The Panel wishes to draw attention to the fact that cross-European research in the social sciences is 
at a very early stage of development. Therefore, TSER thematic networks and concerted actions 
should be particularly advertised because they are adequate means to mobilise the social science 
research community in Europe. 
4. The Panel identified an urgent need for an overview of socio-economic research done in the 
member states, in the Commission and in other specific programmes. This issue should be 
discussed in the Ad Hoc Advisory Group on Co-ordination for the Targeted Socio-economic 
Research Programme. 
1.2.1.2 Future of Community activities in the field of socio-economic research 
5. Since many of the major strategic issues facing the EU over the coming years have socio-economic 
aspects, and since this area of the social sciences is so far underdeveloped in Europe, the Panel 
stresses the crucial importance and pioneering role of a Community socio-economic research 
programme. The Panel strongly supports the idea of a dual approach: to have a separate well 
designed and coherent specific socio-economic research programme in the 5th Framework 
Programme underpinning research and study activities by other specific programmes and 
Commission services, and to further develop socio-economic research as parts of specific 
programmes where appropriate. 
6. To develop solutions for the most pressing European challenges calls for system-oriented 
approaches building on multi-dimensional perspectives, where the social dimension is essential. 
These issues are particularly important in the light of the tensions that are building up in the 
connection of deepening and enlargement of the European Union, which have the potential to place 
barriers to further realisation of the goals of the Maastricht Treaty. The social sciences have to 
make important contributions both from the point of view of their broad range of special 
disciplinary aspects and their ability to develop appropriate social processes and instruments for 
shaping the future. Since the problems are truly European in nature they have to be dealt with 
collaboratively at European level. 
7. In preparing the future programme not only the definition of contents is essential but also the 
design of an appropriate, well balanced set of instruments and modalities for participation: 
Thematic networks, concerted actions, research projects, observatories and accompanying 
measures, especially to ensure continuous communication and interaction with target audiences. 
129 
13. 
8. The preparation of the future socio-economic research programme has to be realised in close co­
operation with the scientific community and potential users of results. 
9. In the preparation of FP5 the Commission should follow an integrative approach, ensuring the 
active support and commitment of all Commission services with a specific interest in socio­
economic research. 
10. The Panel recommends that appropriate means are offered to integrate into the research activities 
groups from those countries from central and eastern Europe who are applying or have already 
applied for EU membership. 
1.2.2 Management and procedures 
11. To better cope with the intellectual aspects of research management at the research/policy interface 
going beyond mere administrative monitoring on the level of project, project clusters and 
programme, more staff with practical experience in social science research, and with capabilities 
and skills in moderating interactive processes will be necessary. 
12. As a consequence of the problematic personnel situation of the TSER programme, the Panel points 
at an urgent need to clarify the staffing situation of the TSER programme and to take respective 
steps to ensure that the programme is equipped with adequate personnel resources for its core 
activities in accordance with the specific objectives and tasks of TSER and within the limits of the 
indicative budget breakdown defined by the Council Decision. A personnel recruitment and 
development strategy should be prepared in the very near future. 
13. The Panel recommends that procedures followed in the evaluation and the final selection of 
proposals are clearly described in the Information Package. The process should be organised in a 
more transparent way, involving independent experts to support and advise the Commission 
services and the Programme Committee in the preparation of the short list. 
14. The Panel recommends that the efforts of the Commission services towards thematic clustering of 
projects concerning the same strategic issues across different sub-areas and between areas should 
be further developed. They will support inter-project communication, integration and co-ordination 
and ensure the future impact of the programme both at the policy level and within the scientific 
community. 
15. The Panel particularly welcomes that Project Progress Seminars are convened by the Commission 
services regularly, approximately every sixth month. Such seminars are appropriate means to 
support networking with the policy makers, other Commission services and Community 
programmes and initiatives, and the scientific community. In addition, also individual projects 
should be encouraged to ensure interaction with possible users of their results - especially in the 
policy area: integrating them in their projects or to develop other adequate means for 
communication, interaction and networking. 
16. The Panel feels that the TSER programme should contribute to the development of Community 
RTD strategies in a well organised and co-ordinated way. Communication with CERT, CREST, 
IRDAC and ESTA are of particular importance. In addition, the co-operation with the Cellule de 
Prospective and the IPTS should be strengthened and further developed. 
17. There is an urgent need for taking a decisive step to activate ETAN. Adequate measures have to be 
taken to tap not only into the results of the TSER projects but also into socio-economic research 
activities of other specific programmes and Commission services, the IPTS, and other actors in the 
field both at the level of the Community and in the member states. 
18. The Panel feels that exchange of information and co-operation between the TSER programme and 
socio-economic research activities in other specific programmes and by other Commission services 
should be strengthened. The TSER programme should have a facilitating and moderating role in 
that context. 
19. Co-ordination and collaboration with COST should be further developed. 
20. The integration of young promising researchers and research teams in projects with well-
established institutions is welcomed and should be further supported. 
21. The Panel is concerned that a very specific issue arises in relation to the derivation, harmonisation 
and dissemination of databases in the socio-economic area. In the mainstream areas of economics, 
the national statistical agencies are the main bodies functioning in this area, and are very well 
130 
13. 
resourced. However, in areas II and III of the TSER programme there is a need to generate new 
quantitative and qualitative data and to harmonise them across a range of member states in a 
context where national agencies are less active. Thematic networks and concerted actions should be 
advertised to support that problem area. Cheap or free access to EUROSTAT data should be 
provided for funded projects, networks and actions. 
22. The social sciences being organised largely in very small units efficient services for better 
information, advice for the preparation of proposals and support for partner search should be 
further developed. In the member states the network of National Contact Points should be further 
strengthened and activated where necessary. In addition, co-operation with other existing networks 
should be developed. 
23. The Panel proposes that at the end of that first assessment round a feedback process is organised to 
draw lessons from the experiences the different panels made. 
131 
13. 
II. COMMISSION'S RESPONSE 
The Commission welcomes the Panel's report as an essential contribution towards the assessment of 
Community research activities in the field of socio-economic research, and for providing detailed 
recommendations regarding the future development of such activities within the 4th and 5th Framework 
Programmes. 
In particular, whilst wishing to recall to mind the highly dynamic nature of policy relevant research, the 
Commission notes the opinion of the Panel that the rationale underpinning a specific programme in the 
field of targeted socio-economic research is fundamentally sound and that the Programme's objectives 
remain not just valid but also highly pertinent. 
In this context, it is noted with satisfaction that in the opinion of the Panel a number of high quality 
research projects and thematic networks, with the potential of generating considerable added value, have 
been selected following the first call for proposals launched under the TSER Programme; and, that the 
Panel considers that a number of these activities are already producing interesting and policy relevant 
results. 
As regards the preparation for the forthcoming third call for proposals for research actions the Panel 
recommends that the work programme of the TSER Programme should be revised in order to restrict 
coverage to a number of key strategic themes. The Commission also foresees the need to focus on key 
issues and will consider the recommendation to revise this work programme whilst respecting also the 
need to take into consideration the outcomes of the first and, the recently launched, second calls for 
proposals. 
Moreover, and in the context of the medium-term development of socio-economic research carried out at 
the European level, the Commission welcomes the Panel's recommendation that social sciences should be 
developed within the 5th Framework Programme through a well defined social sciences research 
programme addressing a coherent set of social challenges facing Europe from a policy oriented 
perspective, as well as being developed as an integrated part of other specific programmes where 
appropriate. 
Finally, with respect to the Panel's other recommendations and in particular those concerning the 
operational and management aspects of the Programme, the Commission wishes to commend the Panel on 
its objectivity and frankness. The Commission undertakes to address these issues and its services will 
report on the measures to be taken as part of the annual monitoring exercise. 
132 
13. 
III. EVALUATION PANEL 
Manfred Horvat (Chairman) 
1993 - : Director, BIT (Bureau for International Research and Technology Co-operation); Member of 
INNOVATION and TMR Programme Committees; Member of ACTS Monitoring Panel 1995 and 1996; 
1995 - : Vice-President, SEFI (European Society for Engineering Education); 1993-94: Visiting Fellow, 
University of Oxford; 1985 - : university lecturer in Technology Assessment and Environmental Impact 
Assessment (Vienna University of Technology, University of Linz); 1980 - : Director (at present on 
leave), University Extension Centre, Vienna University of Technology (University-industry technology 
transfer, technology assessment, continuing engineering education, management of international 
programmes and projects in research and education) 
Prof Francisco Alvira-Martin 
Professor of Sociology, head of Department of methodology and research techniques, Universidad 
Complutense, Madrid; he specialised in programme evaluation as a visiting scholar in North-Western 
University with Prof R.B. Boruch and has written two books and more than five articles on the evaluation 
research methodology; actually engaged in the evaluation of the Drug Abuse Plan of Madrid; 1995: 
independent expert, panel that wrote the draft for the Evaluation Indicators System of the Spanish National 
Science Plan. 
Dr John Bradley 
Economist and Research Professor, Economic and Social Research Institute, Dublin; IRL; specialised in 
labour market economics and the economic processes of cohesion within the EU periphery 
Prof Pierre Desmarez 
Université libre de Bruxelles; Doctor in social sciences; Director, Centre de sociologie et d'économie 
régionales; research topics: work and (un)employment, work histories, transitions from school to work 
Dr Luigi Felici 
1986 - : Director, ANCE, IT (strategic programmes in the field of urban renewal market, economic 
analysis of infrastructure investments, R&D programmes in construction industries); before 1986 L. Felici 
had a leading role in Confidustria's industrial R&D programmes, Mezzogiorno industrial development 
strategies, environmental problems in industry, industrial policy options, external affairs and 
communication); 1979 - 94: member of FAST/MONITOR Management Committee, nominated by Italian 
Research Ministry 
133 

14. 
1996 FIVE-YEAR ASSESSMENT 
COOPERATION WITH THIRD COUNTRIES 
135 

14. 
I. EXECUTIVE SUMMARY 
A. Brief history of the INCO programme and scope of the Five-Year Assessment 
The Second Activity of the Fourth Framework Programme (FP4) is implemented by a specific programme 
of research and technological development including demonstration (RTD) with third countries and 
international organizations. This specific programme, adopted by the Council of Ministers on 23 
November 1994 and known as the INCO (International Cooperation) programme, brought together for the 
first time all EU RTD cooperation with third countries with the exception of the nuclear field. The INCO 
programme is complemented by access to the specific programmes of the First Activity for all European 
and some non-European participants in accordance with general rules established by the Community. The 
cooperation now contained within the INCO programme was previously carried out through a number of 
different programmes. In developing countries three different instruments of scientific and technological 
(S&T) cooperation were replaced by the single programme INCO-DC. In 1991 the Community embarked 
on a major expansion in RTD collaboration with the countries of Central and Eastern Europe (CEE) to 
support the integration process and in 1992 launched the PECO/COPERNICUS programmes which were 
later extended to the New Independent States (NIS) of the former Soviet Union. Collaboration with non-
European industrialized countries has developed in a variety of ways reflecting mutual interest within or 
outside S&T cooperation agreements. INCO is a horizontal programme which cuts across technical fields 
and incorporates the Community's linkages with COST, EUREKA and international scientific 
organizations. 
This five-year assessment covers the activities and management of the INCO programme and its 
predecessors during the period 1991-early 1996 and incorporates the final evaluation under the Third 
Framework Programme. "The four main geographical sectors covered by the INCO programme and 
considered in the report are: 
Area Ah Cooperation with other Fora for European S&T Cooperation: 
COST, EUREKA, International S&T Organizations and Institutions 
Area A2: Cooperation with the Countries of Central and Eastern Europe 
and with the New Independent States of the former Soviet Union 
Area B: Cooperation with non-European Industrialized Third Countries 
Area C: S&T Cooperation with the Developing Countries 
The assessment panel has examined in detail the results and progress made in each main sector and 
considered what useful lessons might be drawn from the Programme as a whole. The full report of the 
Panel is available in English as a separate document. 
B. The panel's comments and findings 
1. Increasing globalisation of the economy and the spread of the Information Society are ensuring that, at 
the threshold of the 21st century, international cooperation in science and technology is more intensive and 
widespread than ever before. A high level of international S&T activity is required to strengthen the 
science and technology basis of EU industry and to promote all the research actions necessary to fulfil 
Community policies. The Panel considers that bringing together the various programmes of international 
collaboration within INCO has resulted most importantly in greater coherence and visibility for EU S&T 
international cooperation. 
2. The Panel finds that the programmes implemented in the period under review were generally of 
satisfactory relevance to the objectives laid down. A substantial number of concrete results, most with 
potential for further research, have been obtained and disseminated. Some significant scientific 
achievements have been recorded (Annex 11). A proper assessment of industrial exploitation would 
require a longer time scale but it is not likely to be large. The burdens of management were increased by 
oversubscription, communication difficulties, local conditions in some partner countries and the overriding 
137 
14. 
requirements of accountability and quality. The formal procedures followed by the Commission were not 
conducive to efficiency improvements in some areas. 
3. The Panel was impressed by the results that have been gained by COST and supports the existing 
relationship between COST and INCO. The joint initiative in vaccine research, for example, has 
facilitated long-term cooperation between the scientific research sector and industry. In the recent past 
there have been notable initiatives in the chemistry domain. COST enables R&D collaboration between 
scientific research bodies and industry to take place in a flexible manner as recommended in the 
Commission document Coordination through Cooperation . COST also provides a nursery for projects in 
specific programmes, is a useful vehicle for participants from outside the EU, and has helped to shape 
elements such as ESPRIT and FLAIR in the Community's Framework Programme. 
4. Because of radically different selection procedures and time-scales collaboration between EUREKA 
and the Community's programmes requires exceptional efforts to achieve results but might be improved 
through procedural modifications (see Recommendation 2 in Part C of this Executive Summary). 
5. The Panel considers that both the COPERNICUS and INTAS programmes were essential, temporary 
responses to urgent needs arising in Central and Eastern Europe (CEE) and in the New Independent States 
(NIS). These programmes have helped the ongoing process of stabilising research potential in CEE 
countries and in the NIS and have introduced many CEE/NIS scientists to modern methods of scientific 
research in a competitive environment. The economic impact of these programmes in many CEE/NIS 
countries has been very limited because of local deficiencies in infrastructure and lack of suitable 
mechanisms for exploitation. High priority should now be given to increasing PHARE and TAOS 
support for essential structural reforms in RTD and in the social and industrial applications of RTD. This 
major challenge requires determined coordinated action and a constructive partnership with the CEE/NIS 
authorities before INCO objectives in the economic, social and environmental sectors can be attained. The 
potential benefits for the whole of Europe are large. 
6. Progress towards participation on an equal footing in the First Activity by CEE research teams 
comparable in standing with western groups should be encouraged by the EU whenever this has a realistic 
basis. A separate adhesion fund outside the Framework programmes will be required to support this 
participation. The transition to fully effective research partnership can be accelerated by phasing out the 
COPERNICUS programme by the end of FP4. 
7. The NIS, which retain a wealth of scientific expertise, particularly in Russia and Ukraine, require a 
different approach. The useful experience built up by INTAS in establishing co-funded research 
programmes provides a basis for continuing collaboration with important research centres, including case-
by-case participation in specific programmes. 
8. Despite intensive efforts by the Commission to publicise EU collaboration programmes within the CEE 
countries and the NIS there is still a very low level of CEE/NIS participation in other specific programmes 
and widespread lack of understanding among individual scientists of the procedures and opportunities 
available. During the transition period these efforts need to be strongly reinforced through all available 
information channels by very clear and unambiguous statements of eligibility for participation in different 
programmes with practical details of costing and payments procedures. 
9. The Panel considers that collaboration with non-European industrial countries and emerging economies 
is rapidly growing in importance. It supports the views expressed in the 1995 IRDAC report that all 
Community R&D programmes should be open to participation by organizations from outside Europe on a 
case-by-case basis provided that European industry in general, and competitiveness in particular, can 
benefit and that reciprocity and suitable IPR arrangements can be established. Emerging economies, 
which are currently eligible to participate in the developing countries programme, should also be included 
under the same conditions in respect of fields where they have achieved high S&T standards and 
performance. The Panel agrees with the Commission's current policy of negotiating S&T cooperation 
38 
14. 
agreements with foreign governments whenever, on the basis of wide consultation with representatives of 
science and industry, there is strong evidence of mutual interest between the prospective partners. 
10. The original aims of the Japan S&T Fellowship Programme have largely been accomplished and, 
taking account of existing bilateral schemes, the Panel considers that this Programme should henceforth be 
focused on co-funded and Japanese-funded arrangements for carefully selected candidates and host 
research laboratories (university, national and industrial) of the highest quality. 
11. The Panel has been impressed by the high reputation of the STD/INCO-DC programme among both 
coordinators and participants deriving from the integrity of the evaluation process, the creation of 
successful partnerships and the applicability of the research projects. The current practice of linking 
postgraduate training (leading to a higher degree) to INCO-DC research projects should continue. The 
Panel considers that the quality of peer reviewing is a most valuable asset of INCO-DC and that it is in the 
strong interest of Europe to be seen to practise even-handedness. 
12. While INCO's evaluation procedures are generally well regarded they are expensive in terms of 
resources. The contribution of information and communications technologies to the submission and 
evaluation of proposals has yet to be proven but the Panel considers that the potential benefits are large for 
all INCO-administered programmes. 
13. The Panel considers that globalisation is often driven by short-term economic factors and by the 
prospect of access to cheap labour in developing countries. INCO has an important role in ensuring that 
basic and applied sciences are not forgotten, particularly in matters that relate closely to regional issues 
and problems. 
14. In the opinion of the Panel the low level of direct industrial participation in the INCO programme 
arises from a combination of thematic and structural causes. The IRDAC report of October 1996 contains 
recommended actions which should lead to a greater involvement of industry in the cooperation 
programme and to the creation of links that are favourable to innovation. 
15. The breadth and complexity of the INCO programme distinguish it from most other specific 
programmes and impose considerable management demands on Commission staff. Constant and close 
collaboration is necessary between DG XII-B and professional staff in other directorates of DG XII and in 
other DGs dealing with individual programme elements. In the view of the Panel the importance of the 
INCO programme requires a strengthening of the role of the INCO Programme Committee and a 
considerably higher publicity profile. 
16. In considering coherence between EU and national S&T policies the Panel has come to the view that 
INCO has a particular contribution to make in problems of a truly regional dimension which are not 
normally tackled by bilateral programmes and for which scientific networking is of paramount importance. 
INCO programmes can usefully complement national programmes by setting up European research 
networks on priority topics and through interdisciplinary regional projects containing both theoretical and 
applied research and requiring a balanced EU/ third country input. Coherence requires adequate 
knowledge on both sides and an effective two-way flow of information. The Panel supports the INCOPOL 
study which aims to collect and synthesise information on national activities in external R&D cooperation. 
C. The panel's recommendation 
COST. EUREKA. International S&T organizations 
1. In view of the potential of COST for future expansion, its high relevance to EU policy and beneficial 
interaction with the Framework Programme, an increase in resources for the COST Secretariat should be 
considered and measures taken to simplify the organizational and administrative procedures as far as 
possible. 
139 
14. 
2. The scope for expanding cooperation between EUREKA and Community projects should be explored. 
For this purpose a pilot scheme providing fast-track procedure for Community participation in support of 
the research part of labelled EUREKA projects should be established with limited fund allocation and 
duration. The outcome of this scheme should be used as an input to determine the extent of further support 
aimed at strengthening collaboration between the Community and EUREKA. 
3. Without evidence of clear added value the Panel considers that the limited activity between INCO and 
international S&T organizations does not at present justify a separate category within the INCO 
programme. 
CEE countries and the NIS 
4. Moves in CEE countries and the NIS to reform the S&T sector need to be accelerated and reinforced, 
through close collaboration between INCO, PHARE/TACIS and the countries concerned, in three main 
directions: 
• restructuring actions aimed at concentrating S&T at a small number of centres of excellence with 
advanced equipment and high-calibre scientists; 
• assisting in the reorganization of S&T administration, giving emphasis to modern transparent methods 
of science management, competitive grants and peer review, and complementing what is already being 
done bilaterally; 
• encouraging the application of S&T in society and the economy through joint efforts in tackling socio­
economic problems. Particular attention needs to be given to IPR protection as a means of stimulating 
innovation. 
5. The COPERNICUS programme in CEE countries and in the NIS should be phased out by the end of 
FP4. During the remainder of the COPERNICUS programme much stronger focusing of calls on 
priorities not covered by other specific programmes, particularly of a regional nature and established in 
negotiation with R&D funding agencies, should be put into effect together with the transfer in appropriate 
cases of the scientific coordinator's role to a CEE/NIS scientist. Following the end of the COPERNICUS 
programme funds should be made available for more generous support for networks, conference 
attendance, modernisation of equipment at selected centres, and other measures so that the amount of 
contact remains at the high level sought by both eastern and western scientists. 
6. The function and objectives of INTAS should be urgently reviewed so that it is able to undertake a 
revised role in FP5 in accordance with the need to help restructure the S&T sector in the NIS, in 
collaboration with TACIS and the countries concerned, intensify high-quality collaboration with NIS 
scientists, continue co-funding arrangements and promote NIS participation in the First Activity. A joint 
analysis with the Russian authorities of important concentrations of civilian research outside Moscow and 
St. Petersburg would help to increase the potential for mutually beneficial collaboration. INCO should 
also identify, together with TACIS, reform-minded funding agencies in the other NIS countries with a 
view to setting up collaborative links along the lines of the relationship between INTAS and the RFBR 
(Russian Foundation for Basic Research). 
Developing countries 
7. The original level of funding S&T collaboration with developing countries (DCs) should be restored in 
view of the high relevance of the results of INCO-DC to current needs, the growing importance of S&T to 
solving regional problems (with world-wide implications), and the need to fund a substantially greater 
number of A-rated proposals and sustain the results already achieved. 
8. The basic objectives of INCO-DC and the major development areas of health, agriculture and 
environment should be maintained. To reduce oversubscription these areas should be more strongly 
focused. The fourth area technology issues, which benefits not only industrial development but adds 
value to all the cooperation programmes, should be given higher priority so that the benefits resulting from 
combining technologies and from Information Society trends, in particular, can be made more widely 
accessible in developing countries. 
140 
14-
9. High priority should also be given to INCO collaboration with EU programmes that support 
infrastructure developments, such as ALA, MEDA and LOME, in order to raise awareness of this issue, 
create stronger links between infrastructure investment and supporting RTD projects, and encourage 
appropriate actions in developed countries. 
10. In the light of requests from the developing countries the role of DC participants in setting priorities, 
formulating and evaluating proposals, and in the scientific management of projects should be strengthened 
and the possibility of entering into co-funded arrangements should be examined. 
Evaluation 
11. In order to make the evaluation procedures more transparent and help newcomers to Community 
research programmes learn through participation in the application process, prompt notification should be 
given to proposers of unsuccessful projects together with reasonable feed-back on their rating. 
12. A pilot trial on the use of information and communications technologies in the submission and 
evaluation of proposals should be carried out in a future call to determine whether the quality of the 
preliminary stages of the evaluation process, including the necessary confidentiality, can be preserved with 
a view to preparing the ground for shorter, more focused meetings in Brussels. 
EU and national S&T policies 
13. Continuous monitoring of RTD actions on a bilateral basis between Member States and countries 
participating in PECO/COPERNICUS and INCO-DC programmes is desirable in order to adhere to 
subsidiarity and avoid unnecessary duplication. 
INCO Programme Committee 
14. The mandate of the INCO Programme Committee should be carefully examined and revised where 
necessary in order to strengthen the Committee's role in the development of a visible and effective INCO 
programme and to allow time for the Committee to consider matters of a strategic and long-term nature. 
The Commission, with the help of members of the INCO Programme Committee in their own countries, 
should make strenuous efforts to raise the profile of INCO by widely publicising its aims and objectives 
and the opportunities it provides for scientists to engage in international collaboration on relevant issues. 
Administration 
15. The Panel strongly recommends that the Commission give urgent consideration to the introduction of a 
comprehensive Management Information System for the INCO programme capable of giving at any 
moment a bird's-eye view of main trends and indicators and the status of projects. This management and 
monitoring tool would facilitate coherence among all DGs actively concerned with the programme, and 
enable the INCO Programme Committee and future monitoring and assessment panels to focus rapidly on 
critical issues. Since some areas of the cooperation programmes appear to lack effective monitoring and 
reviewing procedures for ongoing projects a system of mid-term monitoring on a sampling basis, including 
independent financial auditing, should be implemented forthwith and publicised. 
16. In order to streamline the administration processes within the Commission, a subject of continuing 
concern to both coordinators and participants, approval for projects within a defined ceiling should be 
given fast-track treatment. Consideration might be given to modifying the existing delegation and 
signature procedures within the Commission consistent with the need to maintain transparency. The 
Community cannot afford to show inefficiency in its own operation and at the same time urge reforms on 
its DC and CEE/NIS partners. 
Assessment and monitoring 
17. The Panel considers that in the five-year period under review the INCO programme and its 
predecessors were necessary to meet the circumstances of the time and have substantial achievements to 
their credit. In a rapidly changing world there will be a continuing need for adjustments within the four-
year period of a Programme. In regard to the CEE countries and the NIS, the emerging economies and 
the developing countries there will be many new challenges in the coming few years. In this situation the 
141 
14. 
Panel recommends that both the monitoring and assessment exercises on the lines put forward by CREST 
should remain essential components of the INCO programme. 
142 
14. 
II. COMMISSION'S REPONSE 
The Commission welcomes this independent report and appreciates the effort which had to be made by 
Panel members to come to grips with such a complex programme in such a short period. The report 
underlines the importance of a high level of international S&T activity to strengthen the S&T basis of EU 
industry and to promote all the research actions necessary to fulfil Community policies. 
The Panel recognizes the importance of the INCO Programme in providing coherence and greater 
visibility to EU S&T international cooperation, and points out the particular contribution which can be 
made by INCO to the solution of problems of a regional dimension which are not normally tackled by 
bilateral programmes or other specific programmes. 
The Commission agrees on the potential of COST for future expansion, its high relevance to EU policy 
and beneficial inter-action with the Framework Programme. Any increase in resources will however need 
to be carefully balanced with the needs of other parts of the Framework Programme. A High Level 
Reflection Group has been constituted to consider the future role of COST in the European S&T 
landscape. Maximum opportunity will be taken to simplify and stream-line the organizational and 
administrative procedures, consistent with the Community's financial regulations. The detailed measures to 
be worked out will draw substantially on the results of the COST evaluation now taking place, in relation 
with the COST Senior Officials and the COST Ministerial Conference in May 1997. 
Concerning the suggestion of a pilot scheme providing fast-track funding for Community participation in 
Eureka projects, any reservation of funds specifically for EUREKA projects does not seem realistic. 
However, the Commission intends to follow up the suggestion to explore means whereby urgent 
EUREKA, and of course other relevant projects could nevertheless be catered for, as has been done in the 
past (e.g. COSINE, JESSI). 
The Commission considers that a clear point of reference responsible for international organizations is 
needed to deal with the multitude of S&T issues regarding them 
With respect to Cooperation with Central and Eastern European countries, the Commission agrees that, 
particularly in the case of the pre-adhesion countries, full participation in Activity 1 programmes is the 
ideal situation and should indeed be encouraged, it should nevertheless be recognized that participation in 
these programmes through FP4 has been very low so far. Conversely, the oversubscription to the INCO-
COPERNICUS programme has been in the order of 10 to 1. This is due to some extent to the different 
funding regimes of Activity 1 programmes and COPERNICUS, however it also indicates that while there 
is a great desire for cooperation between Eastern and Western scientists, there remains a strong 
requirement for a dedicated programme. The Panel recommended that COPERNICUS be phased out with 
the end of FP4 and that participation in Activity 1 be encouraged "whenever this has a realistic basis". For 
most CEECs, however, such a realistic basis does not yet exist and full participation will only gradually be 
achieved. The "accompanying measures" recommended in point 5 would go some way towards filling the 
gaps left by COPERNICUS but probably not far enough. It should also be remembered that one of the 
important roles of INCO, recognized by the Panel itself, is its contribution to the solution of problems of a 
regional nature, problems which are not specifically addressed by Activity 1 programmes. In preparing for. 
FP5, the Commission intends to increase the focus of the specific RTD cooperation with CEECs to 
concentrate on the issues of particular relevance to these countries, as well as placing more emphasis on 
accompaning measures, as suggested by the Panel. 
The Commission does not share the view that the role previously played by INCO-COPERNICUS in the 
NIS should be completely taken over by INTAS, including the aspect of financing the participations in 
Activity 1. The role played by INTAS in S&T cooperation with the NIS is an important one, but we do 
not consider that extending it further would lead to any improvement of the situation concerning 
participations in Activity 1. To a certain extent, INTAS and COPERNICUS address different types of 
activity. While INTAS is mainly devoted to fundamental research, COPERNICUS has a more direct 
143 
14. 
approach towards application oriented and industrial research. For the management of financing 
participation in Activity 1 programmes, no special agencies would be needed as this is already established 
within the Commission services of the relevant programmes. In the preparation of FP5, it is intended to 
look closely into the possibilities for support of Centres of Excellence in certain CEC/NIS, as 
recommended by the Panel. The Commission agrees with the importance placed by the Panel on the close 
cooperation necessary between PHARE/TACIS and INCO and it is intended that a joint communication 
(DGIA-DGXII) on S&T cooperation with the pre-adhesion countries be produced in 1997. 
The Commission notes the support of the Panel for its policies in negotiating S&T cooperation agreements 
with non-European industrialized countries including Emerging Economies and shares the Panel's view 
that reciprocity and suitable IPR arrangements should be established and that collaboration be implemented 
through the specific programmes on a project-by-project basis. The general need to review the structure 
and operation of the Japan S&T fellowship programme is acknowledged. However, the proposed phasing 
out of the EU fully financed fellowships would result in eliminating the access they provide to Japanese 
industrial laboratories, since the other schemes, which involve Japanese funding, are restricted to public 
research organizations. The Commission welcomes the suggestion of integrating into a more general 
fellowship programme the Korean and any new geographically oriented fellowship scheme, which may be 
instituted with countries in Area B. 
With respect to cooperation with Developing Countries (Area C), the Panel has underlined the importance 
of this part of the programme and recognized that the major development areas of health, agriculture and 
environment should be maintained. The Commission shares the opinion that applied research in the 
technology area is important and that additional funds will be needed in order to adequately tackle this 
issue. As a first step, it is intended to increase the funding available to "Additional sectors of mutual 
interest" in the INCO-DC programme of FP4 from 20 MECU to 30 MECU. Specific cooperation policies 
are also being developed for those Developing Countries which, due to their economic and technological 
growth, should be considered as emerging economies. While the Commission appreciates the large 
number of very highly rated proposals submitted in the context of INCO-DC, it recognizes the necessity to 
reduce the oversubscription and thus the wasted effort on behalf of the scientific community. 
Concerning the recommendations relevant to all programmes, the Commission agrees that it is necessary 
to constantly make efforts to reduce perceived "procedural inefficiencies" leading to long delays between 
submission of proposal and the first transfer of funds to the successful applicants and to ensure that 
unsuccessful applicants are informed rapidly of their situation. Continuous attempts are being made to 
ameliorate the internal Commission administrative procedures while remaining consistent with 
Commission's duty of public accountability. 
The electronic submission of proposals has already been tried by some programmes in the past. The 
possibilities could be examined again in the light of updated electronic methodologies. The facilities 
available in relevant countries would need to be investigated and to be taken into account. 
The Commission fully agrees with the recommendation that an integrated Management System be 
introduced. Although the suggestion is relevant to all programmes, it is particularly so in the case of 
INCO, where several different DGs are involved in its management. It is intended to improve this situation 
as far as possible over the remainder of FP4, and particularly to have a system in place before the 
commencement of FP5. 
Concerning the role and mandate of the INCO Programme Committee, it is governed by the Comitology 
Decision of 1987 and Article 130i of the Maastricht Treaty. 
144 
14. 
III. EVALUATION PANEL 
Mr Luis Avelar 
Director, Portugal Telecom 
Prof. Niels Ole Bernsen 
Center for Cognitive Science 
Roskilde University 
Denmark 
Dr.Stein W Bie 
Director, Research, Extension and Training Division, 
Sustainable Development Department 
Food and Agriculture Organization of the United Nations 
Mr Christian Dambrine 
Association Nationale de la Recherche Technique, France 
Member of the Bureau, European S&T Assembly 
Prof. Dervilla Donnelly 
University College Dublin, Ireland 
Vice-President, European S&T Assembly 
Dr Désirée Edmar 
Director, Naturhistoriska Riksmuseet 
Swedish Museum of Natural History, Stockholm 
Mr Terence Garrett 
Formerly Department of Trade and Industry 
and the Royal Society, UK 
Dr Werner Menden 
Formerly Bundersministerium fur Forschung und 
Technologie, Germany 
Member of the 1995 Monitoring Panel for the INCO Programme 
Dr.Alberto Silvani 
Consiglio Nazionale delle Ricerce 
Italy 
145 

15. 
1996 FIVE-YEAR ASSESSMENT 
DISSEMINATION AND OPTIMIZATION OF RESULTS 
(INNOVATION) 
147 

15. 
L EXECUTIVE SUMMARY 
1. Among European decision-makers, there is general agreement that frequent and systematic 
innovation is a main source of new jobs, high productivity, and, in the end, of wealth creation. But 
innovation needs a favourable environment. This means management skills, circulation of 
knowledge across countries and sectors, flexible product markets, and market-oriented RTD. It also 
means standards and regulations that promote innovation; a tax policy conducive to innovation; 
conditions favouring the development of capital markets for financing innovation; a European patent 
policy that facilitates access to intellectual property rights and defends those property rights more 
effectively worldwide. Finally, much more than is now the case, research institutions and industry 
should work together to meet the needs expressed on the market by users and consumers. 
2. INNOVATION, the specific programme corresponding to the Third Activity of Framework 
Programme IV, deals with most of this. INNOVATION became operational following the Council 
Decision 94/917/EC of 15 December 1994. The programme has a budget of 312 MECU. It carries 
on activities started under SPRINT and VALUE II, and one action line from THERMIE. Its basic 
mission is to strengthen the innovative capacity of European industry. The programme must also 
contribute to economic and social cohesion. There is a special emphasis on SMEs and technology 
diffusion and circulation. All the programme must be conducted under the principle of subsidiarity: 
it must complement, but not be a substitute for, national and regional activities in similar domains. 
3. The INNOVATION programme is evolutionary and experimental. It is particularly important, 
therefore, that it be regularly refocused by proper monitoring and evaluation based on verifiable 
objectives. The time and resources available for this report, however, have dictated that it be more a 
review on the basis of relevant documentation, some interviews, and our general knowledge of the 
area, rather than an evaluation with strong empirical backing. The Commission should adopt 
measures to ensure more effective evaluation in future. 
4. The Terms of Reference for this assessment included five questions. We were asked to judge the 
relevance, effectiveness, and cost efficiency of the initiatives carried out within the domains of the 
INNOVATION programme in the years 1991 to 1995. Moreover, having in view the socio­
economic conditions expected pver the five years to come, we had to make recommendations about 
the future orientation and management focus of the programme. The assessment had to take into 
account the final evaluations of VALUE II, which was part of Framework Programme III, and of 
SPRINT, which was outside the Framework Programme. 
5. Our overall impression is positive. INNOVATION and its precursors SPRINT and VALUE II 
have addressed issues relevant to the Community's socio-economic needs. These programmes have 
increased the understanding of the innovation processes and policies within the Commission and 
among national and regional actors. The merger of SPRINT and VALUE II was a necessary step to 
rationalize previously separated streams of action. There are some priorities to reassess and some 
shortcomings to fix, but these problems are not paramount. 
6. In particular, INNOVATION and its precursors have: 
• Set up infrastructures to disseminate information on the Community RTD activities; 
• Learned and disseminated useful lessons on the innovation processes in SMEs; 
• Promoted horizontal actions within the Commission to make RTD and other Community policies 
more innovation oriented. Regional actions are the most successful example; 
• Created a seed for a European innovation policy. The Green Paper on Innovation should get much 
of the credit for this; 
• Recognized that action at Community level should complement national and regional activities, not 
be a substitute for these. 
7. We believe that innovation policy should have an enhanced role in EU activities. We emphasize 
in our report the importance of local connections and even personal contacts in diffusing innovative 
ideas. We believe, however, that there is a place for Community action, provided that excessive 
centralization is avoided. The action plan based on the Green Paper on innovation should lead to a 
European innovation policy with a distinct and strong role among the other EU policies and among 
national and regional innovation policies. This policy should be supported by new organizational 
arrangements within the Commission. 
149 
15. 
8. In the light of this, it would be right to give INNOVATION a strategic and more influential 
role. INNOVATION has been hampered by having too few resources and too little authority. It has 
nevertheless made progress in the direction of a European innovation policy. It therefore deserves to 
be enhanced. 
9. INNOVATION should be a wide-ranging and flexible entity within the Framework 
Programme. There is a complex relationship between RTD and innovation. Each contributes to the 
other, but innovation is a wider concept and has other roots, besides those in RTD. On balance, we 
feel it is right to keep INNOVATION within the Framework Programme. The Framework 
Programme, however, must not unduly constrain INNOVATION from its wider activities. This 
means developing a complex of horizontal interactions and may need a rather more open structure 
than is appropriate for the RTD programmes. As we mentioned before, the programme must be able 
to experiment and evolve. This, in turn, means there must be well defined procedures for initiating 
and terminating activities, i.e. 'entry and exit mechanisms'. This also implies that the programme 
must be able to respond to needs that change rapidly. Flexibility should be increased. 
10. We would like to see INNOVATION and its staff acting as a 'ginger group' within the 
Commission as well as outside it. A 'ginger group' is a small group of people whose role is to 
provoke and stimulate activity, to encourage new thinking and hence to enliven a larger group. This 
is precisely the part we think those involved with the programme could and should play. By 
disseminating ideas, new experience, and the results of experimentation, they would provoke and 
stimulate innovative approaches. This would give the programme staff something of the nature of a 
think-tank for pursuing the aims of the Green Paper. They would need to maintain a clear view of 
how innovation works, provide a strong element of reflection and experimentation, and have the 
main objective of promoting innovation inside and outside the Commission. 
11. We recommend strengthening the programme's coordination with other EU policies and DGs. 
Interfacing with, and contributing to the policies of, other DGs would be a vital element of this 
'ginger group' role. In particular, the programme should be linked with the assessment and 
implementation of activities under Community policies other than RTD, such as the industrial, 
SME, and regional policies: all this to help innovation, as well as technology transfer and diffusion. 
These matters require a committed advocate, but also proper administrative arrangements. 
12. It is particularly important to enhance the interaction between the Third Activity and the RTD 
programmes of the First Activity. In our view, the INNOVATION programme's mandate does not 
distinguish adequately between exploiting the results of specific RTD projects and promoting 
innovative thinking and practices more widely. Within Community-supported research, we think 
there should be greater attention to exploitation, but this should be the responsibility, primarily, of 
the RTD programme involved. INNOVATION, however, should not be a mere downstream service 
available to the RTD programmes on request. It should provide them with guidelines for 
dissemination and optimization and play an interactive role of monitor and mentor throughout their 
whole life. In general, it should increase the orientation of these programmes towards innovation. 
INNOVATION has launched a number of specific actions to ensure a coordinated approach to 
dissemination and optimization. When fully operational, these initiatives could make good progress 
in the direction of effective coordination. INNOVATION should have the authority and institutional 
instruments necessary to ensure that this occurs. 
13. The programme will need to be clearer about its priorities. It will also need a simpler 
structure. All this to give a sharper focus on some issues, increase flexibility, and improve 
internal coordination. We draw attention to the following points: 
13.1 The European Innovation Monitoring System (EIMS) should be a particularly important part of the 
INNOVATION programme facilitating, as it already does, the integration of Community activity 
with that of member states and other organizations. It could play an important role within the 
'ginger group' activities we advocated above for the programme. 
13.2 The programme should expand and intensify its regional actions. In particular, using existing 
initiatives such as the Regional Innovation Strategies or new, more specific, instruments, the 
programme should help eligible regions to make efficient use of structural funds in domains affected 
by technological innovation and should encourage SMEs in these regions to participate in 
cooperative innovation projects. 
150 
15. 
13.3 The INNOVATION Relay Centres (IRCs) are in the best position to integrate action at the local and 
European level. The IRCs should strengthen this as their prime role within the programme. The 
IRCs should work on promoting innovation in their local environments rather than concentrate on 
disseminating information about Community RTD. Whenever useful, they should give a European 
orientation to the demand for innovation and innovation financing of local firms. They could also 
provide valuable services to other parts of the programme, such as the regional actions. To do this 
they need to be aware of all aspects of innovative thinking and to be very close to their local 
economies, building on, but not duplicating, other locally based activities. They also need to be 
close to the Commission and to other IRCs so that they can operate as an effective network. 
13.4 Creating networks is in our view an important role for innovation policy at European level. Through 
these, information and ideas will flow, enabling cross-fertilization of regional and sectoral 
experience to occur and stimulating competitive advances. Such networks will sometimes be 
electronic, sometimes paper-based, and often working through personal contact. They may be either 
permanent, i.e. infrastructural, elements of policy, or experimental initiatives meant to continue on 
their own. Within the INNOVATION programme, networking initiatives other than the IRCs should 
be experimental. They should be designed to become in due course financially independent from the 
Commission. These initiatives could add a European dimension to networks that already exist or 
start cooperation in some new areas where the Commission has competence. 
13.5 We attach particular importance to stimulating a financial environment favouring the diffusion of 
new technologies. So far, however, the INNOVATION programme has made only limited 
contributions to thinking in the field. It should aim at a larger role. Europe has been too slow in 
developing systems for financing innovative activity. These need to be developed in a coordinated 
way to integrate actions at regional, national, and Community level. 
13.6 The programme should retain the important information-providing role which it discharges via 
CORDIS (preferably, further enhanced). It should give more attention to understanding, in depth, 
the needs of users of the service so that the innovation message is broadcast to a receptive audience. 
13.7 We strongly recommend additional effort and a more coherent strategy on patents. This means 
primarily strengthening the present role of support to the RTD programmes. But the development of 
provisions for intellectual property rights is also a strategic issue for a European innovation policy. 
The current situation is far from satisfactory. Measures in this area should be intensified in future. 
13.8 The purpose of projects within the INNOVATION programme should be to demonstrate and 
disseminate innovative approaches and ways in which technology can enhance innovative attitudes. 
It should not be, only, to exploit specific results. These projects should, primarily, serve as 
showcases for displaying innovative thinking and practices generally. They should aim, rather than 
at specific technologies, at the wider objectives of a European innovation policy such as an easier 
access to product markets for new technology-based firms, the increase of the technology absortion 
capabilities in traditional sectors, a more frequent transfer of technologies between sectors and 
countries (not only within the Community, but also across its borders), the stimulation of a 'culture 
to innovate' throughout Europe. 
13.9 We believe that the transfer and validation projects should be merged, the OPET network should be 
transferred to THERMIE, and two of the present action lines, innovation management techniques 
and public awareness of research and technology, should not be individual elements of a future 
programme, although their specific goals may be important. The awareness activities could become 
a strand of the EIMS and be redirected to a coherent promotional strategy of the Community 
initiatives in research, technology, and innovation. The innovation management initiatives could be 
shared out: the EIMS would assess new techniques, whereas the IRCs would promote state-of-the-
art practices. 
14. The previous recommendations may be summarized in the six points below: 
• Innovation policy should have an enhanced role in EU activities. 
• INNOVATION should continue, be significantly enhanced, and become a wide-ranging and 
flexible entity within the Framework Programme. It should provide a 'ginger group' to stimulate, 
promote, and champion innovation within the Commission as well as outside it. 
• There should be greater attention to innovation within the RTD programmes. 
151 
15. 
• INNOVATION should assist the dissemination and optimization activities of the individual RTD 
programmes. It should have the opportunity to intervene where these activities appear to be 
inadequate. In general, it should increase the orientation of the RTD programmes towards 
innovation. Proper administrative arrangements are needed to ensure all this. 
• The coordination with other EU policies and DGs should be strengthened. In particular, the 
programme should be linked with the assessment and implementation of activities under Community 
policies other than RTD, such as the industrial, SME, and regional policies: all this to help 
innovation, as well as technology transfer and diffusion. 
• There should be greater attention to ordering the priorities of the programme's individual action 
lines so as to give a sharper focus on the issues highlighted in the previous points. In future the 
programme would consist of eight major elements. 
152 
15. 
II. COMMISSION'S RESPONSE 
The Commission fully supports the emphasis in the recommendations on the promotion of innovation, both 
in-and outside the RTD Framework Programme. 
As a matter of fact the Commission adopted on 20 November 1996, following the wide ranging public 
debate stimulated by the Green Paper on Innovation, the First Action Plan for Innovation in Europe. This 
Plan considers the ability to innovate as a necessity to maintain competitiveness and a precondition for the 
future creation of jobs in Europe. 
It identifies three areas for action : to foster a genuine innovation culture, to establish a legal, regulatory 
and financial framework conducive to innovation, to better articulate research and innovation. Each of 
those areas includes actions which belong to the realm of science, research and technology policies as well 
as actions which typically belong to other policies such as industrial -, enterprise -, and regional policies. 
The Commission also agrees with the recommendations that there should be greater attention to innovation 
within the RTD programmes and that the INNOVATION programme could usefully guide and assist the 
specific programmes in these efforts. In the light of the experience of the Interprogramme Group on 
Dissemination and Optimization, a specific mandate will be considered for the INNOVATION programme 
providing the basis and instruments necessary to ensure that this mentoring and monitoring effectively 
occurs. 
The same applies as regards the recommendation on the coordination with other EU policies with a view 
to promote innovation, technology transfer and technology diffusion. 
Thirdly, the Commission welcomes the proposed re-structuring of the activities covered by the programme 
and the suggested reduction in the number of action lines. In this context it wants to point out that the 
latest call for proposals for Technology Transfer and Technology Validation Projects 1 already has come a 
long way towards integrating both types of projects as indicated by one of the recommendations. 
Finally the Commission has taken notice of the recommendations that the INNOVATION programme be 
clearer about its priorities : exploiting the results of specific RTD projects versus promoting innovative 
thinking and practices more widely. In line with the orientations adopted by the Action Plan for Innovation 
the latter is to emerge as one of the key priorities for future operations 
OJN°C221, 17.09.96, p. 20. 
153 
15. 
III. EVALUATION PANEL 
Luis A. Oro Giral (E), Chairman 
Professor, Department of Inorganic Chemistry 
University of Zaragoza 
Nicos M. Christodoulakis (GR) 
Professor, Department of International Economics and European Studies 
Athens University of Economics and Business 
Heinrich Hôfer (D) 
Head, Department of Technology and Innovation Policy 
Federation of German Industries (BDI, Bundesverband der Deutschen Industrie) 
Sean McCarthy (IRL) 
Managing Director, Hyperion Energy Systems Ltd 
Erkki Ormala (SF) 
Secretary, Science and Technology Policy Council of Finland 
Ministry of Trade and Industry 
Philip V. Hills (UK) 
Professorial Fellow, PREST (Policy Research in Engineering, Science, and Technology) 
The University of Manchester 
Giorgio Tuninetti (I), Rapporteur 
Assistant to the First Vice-President, Business Development 
Corporate Staff, Finmeccanica SpA 
154 
16. 
1996 FIVE-YEAR ASSESSMENT 
STIMULATION OF TRAINING AND MOBILITY 
OF THE RESEARCHERS 
155 

16. 
I. EXECUTIVE SUMMARY 
This is the Executive Summary1 of the Five Year Assessment of the Human Capital & Mobility (HCM) 
and its successor the Training & Mobility of Researchers (TMR) Programmes. The Assessment is 
undertaken as a precursor to the adoption of the TMR Programme under the Fifth Framework 
Programme. It also incorporates the Final Evaluation of the HCM Programme. 
The recommendations made in this Report are essential to the good development of the TMR Programme 
and have the total and unanimous support of the Assessment Panel. It is the Panel's explicit wish to see 
these recommendations fully implemented. 
Europe, as it enters the 21st century, faces a number of major challenges. Its industrial competitiveness 
must be improved and barriers to the flow and development of information and knowledge broken down. 
Ecological and environmental problems must be overcome. The major social concerns of migration and 
racism, health and lifestyles, social exclusion and governance must be faced. The Panel concludes that 
Europe will be better positioned to face theses challenges, if its scientific and technological research 
community is ready to cooperate across discipline, across culture and across regional and national 
boundaries and that a training and mobility programme has a substantial contribution to make in 
developing this cooperation. 
• The TMR Programme is an essential element in facing these challenges and should be continued 
under the Fifth Framework Programme. 
However, any programme must have the capability of transcending purely EU concerns to ensure research 
encompasses the Global dimension of industrial competitiveness and of ecologically sustainable 
development and, in particular, plays a full part in the development and stabilization of Central and 
Eastern Europe. To respond fully to these challenges, some changes are needed in what the Panel 
considers to be a most successful and well run Programme. 
• The Programme needs to move to working with all research disciplines relevant to these 
challenges and developing in them a full European and Global dimension. 
The Programme must also work closely with European research community itself, where it has earned 
great respect for its emphasis on scientific excellence and a "bottom-up" approach. The latter permits 
dynamic new areas outside of the EU specific Programmes to be supported and corrects for the 
misalignment or inflexibility of highly targeted programmes. It is in a sense an insurance policy. 
However, it does cause major problems in over-subscription in parts of the Programme and confusion in 
Programme objectives : is it an insurance policy for the research not covered by specific Programmes or is 
it a training and mobility programme ? 
The Panel recommends that : 
• All 5FP Programmes should have a section for "bottom-up" research - their insurance policy. 
The TMR should remain bottom-up but concentrate on training and mobility issues. 
Turning to the individual Programme Activities : 
• The Fellowships Activity should be developed as three basic activities. The Panel recommends that : 
1. The Marie Curie Fellowship should be developed explicitly for the very ablest of Europe's 
researchers with a proven research record. 
^his Summary is published on a stand-alone basis in all Union languages (Document 1) and as part 
of the full report in English only (Document 2). 
157 
16. 
2. Industrial Host Fellowships should provide support specifically for the training researchers in 
industry. 
3. Structural Cohesion Fellowships should provide specifically for the development of high level 
research teams in Less Favoured Regions. 
• The Research Networks Activity should increase the success rate and value for money of networks 
by: 
1. Increasing the flexibility of network structures funded, possibly moving away from the average 
funding per node basis. 
2. Providing a maximum of 50% funding for projects, 
3. Increasing quality of networks fellows by open, international advertisement of positions, 
4. Encouraging interdisciplinary proposals and ensuring their equitable evaluation structure. 
• The Large Scale Facilities Activity should : 
1. Increase the cluster development and co-ordination elements of the Activity. 
These have a high EU value added. 
2. Broaden their definition to being "Large Scale Intellectual Centres of Excellence". 
3. Avoid becoming the platform for the planning of news LSFs. This might become damaging to 
the main objectives of the Activity. 
4. Be reviewed for repositioning within the Framework Programme. 
• The Accompanying Measures Activity : There is a major need for a "Strategy & Follow-up Unit" to 
support the Activities of the TMR Programme. This Unit will have many tasks essential to overall 
Programme development including developing : 
• An overall strategic and analytic capacity to support the Programme, 
• A working model of the European industrial and academic research community needed for 
Programme planning and evaluation, 
• A follow-up capacity which will coordinate and analyse the impact of the TMR 
Programme, 
• Associated feedback mechanisms to identify shortcomings and potential improvements, 
• A support platform to work with Member States on trends in research training, mobility and 
related issues. 
158 
II. COMMISSION'S RESPONSE 
1. The Commission welcomes the principal conclusion of the Panel that the promotion of the training 
and mobility of researchers is an essential element of the Fourth Framework Programme and should be 
continued under the Fifth Framework Programme. The Commission shares the Panel's view that the 
current Training and Mobility of Researchers (TMR) Programme is a most successful and well run 
Programme. 
2. The Panel has made a number of specific recommendations to improve still further the impact and 
effectiveness of these training and mobility actions, in particular with a view to the Fifth Framework 
Programme. These recommendations will be studied carefully to see how they can best be implemented, 
in particular: 
• focusing the Marie Curie Fellowship scheme on the very ablest of Europe's young researchers 
with a proven research experience; 
• introducing Industry Host Fellowships to provide support specifically for the training of 
researchers in industry, as well as Regional Host Fellowships to improve the development of 
research and research training in the less-favoured regions of the Community; 
• finding mechanisms to increase the success rate and value for money of Research Training 
Networks; 
• introducing Infrastructure Networks as a means of promoting self-coordination among Europe's 
research infrastructure; 
• organising a Strategy and Follow-up Group to provide strategic and analytic support for the 
activities. 
3. The Commission shares the Panel's view that these actions should remain "bottom-up", open to all 
fields of scientific research, and with projects selected on the basis of scientific excellence. 
159 
16. 
III. EVALUATION PANEL 
Professor Daniel Thomas (Chairman) 
Professor at the University of Technology of Compiègne (France). Since 1970, he has held responsibilities 
at a national and European level. He heads the enzyme technology laboratory belonging to the CNRS. He 
has published over 300 scientific papers in international journals. For several years, he was Director of the 
French Biotechnology Programme. At a European level, he initiated the Biotechnology activities in 1978 
and has chaired many E.C. committees. 
Professor Joannes (Jan) Beenakker 
Emeritus Professor of Physics at Leiden University. Before his retirement, he headed the molecular 
physics group of the Department of Physics at Leiden University. His main field of research over the past 
decade has been the study of nonequilibrium behaviour of gases of rotating molecules. He has been guest 
Professor of molecular physics at the Catholic University of Leuven. From 1985 until 1991 he was Rector 
Magnificus of Leiden University. After his retirement, he served as Chairman of F.O.M. (the physics 
section of the Dutch Science Foundation NWO), from 1991 until 1994. He is a member of the KNAW, 
the Royal Dutch Academy of Science and of the Academia Europea. 
Mr. Tom Casey (Rapporteur) 
Director, human Resources, at the CIRCA Group Europe Ltd. and a lecturer in Technology & Innovation 
Policy in University College Dublin. He was educated at Imperial College, University of Manchester and 
Université de Paris-Dauphine. He has worked in the OECD, National Board for Science & Technology, 
National Enterprise Agency in Dublin. 
Professor Peter Hackl 
Professor of Statistics at the University of Economics and Business Administration, Vienna, Austria. His 
scientific interests concern the development and applications of statistical methods for time series analysis 
and for the analysis of economic data e.g. diagnostics for economic and econometric models and methods 
for process control and for quality improvement. He is author of several books and some 70 articles in 
refereed international journals. Since 1995, Professor Hackl has been President of the Austrian Statistical 
Society. 
Dr. Geoff Robinson 
Director of Technology for IBM United Kingdom. He has spent most of his career in Research and 
Development for IBM. From 1992 to 1994 he was Chief Scientific Advisor on Science and Technology at 
the UK Department of Trade and Industry. He is a Fellow of the Royal Academy of Engineering and was 
President of the British Computer Society 1995-96. 
Professor Alexandre Quintanilha 
Holds a PhD is theoretical Physics and is Professor of Biophysics and Dean of the Biomedical Institute of 
the University of Porto. From 1972 to 1990, Professor Quintanilha was Associate Professor of Physiology 
at the University of California at Berkeley and Director for Environmental Studies at the Lawrence 
Berkeley Laboratory. His area of scientific research is molecular markers and mechanisms of adaptation 
under stress in humans. 
160 
17. 
1996 FIVE-YEAR ASSESSMENT 
NUCLEAR FISSION SAFETY 
161 

17. 
T. EXECUTIVE SUMMARY 
The main impact of the programme has been, and continues to be, the provision of a common scientific 
basis for: 
the safe operation of nuclear reactors, 
their effective decommissioning, 
the development of agreed means for the safe management of radioactive waste, and 
the protection of the European population and environment from possible adverse effects of 
radiation, and in particular the past, present or future use of nuclear power and the uses of radiation 
in medicine. 
In reviewing the achievements of the Third Framework Programme (FP3) we have concentrated our 
detailed attention on the Reactor Safety and Waste Management areas, since the Radiological Protection 
and Decommissioning areas have been subjected to independent evaluation during this five-year period. 
In reactor safety, new understanding has been developed of phenomena which may occur in a severe 
accident. Progress was made in the area of core degradation, but uncertainties remain in the so-called 
late phase. Progress was achieved in calculating distribution and combustion of hydrogen, and basic 
knowledge was obtained in the area of molten fuel-coolant interaction. Several interesting core retention 
devices were developed. Fission product behaviour was studied effectively. Accident management 
support was initiated to be available to help in recovery following a severe accident. 
In radioactive waste management, there has been cost-effective use of expensive underground 
laboratories both to develop confidence in safety performance assessment for the selection of future 
suitable sites which will meet national criteria, and to understand mechanisms of migration of 
radionuclides, many of which are relevant also to other toxic substances. There has been useful 
innovation in attempting to reduce the volume of various wastes, the content of long-lived alpha-emitting 
materials to be disposed of in repositories, and in the development of waste forms and containers which 
are stable in the long term. A 'European Network for Quality Checking of Waste Packages' has been 
successful in providing harmonization of quality assurance for radioactive waste products which has been 
of help to national regulatory and licensing organizations. 
Research in decommissioning has decreased as the technology has matured, and the publication of the 
Handbook on Decommissioning of Nuclear Installations has proved an excellent means of disseminating 
results and of stimulating their practical application. 
In radiation protection, the major achievement has been in providing the scientific basis for the 
revision of the EC Basic Safety Standards Directive, which was adopted in May 1996. Fundamental 
advances in understanding DNA repair after radiation damage, the initial events in radiation 
carcinogenesis, and in epidemiology have come from the programme. It has also helped to evaluate 
patient benefit from conventional and newly-introduced radiological techniques and contributed to the 
revision of the Medical Exposure Directive. Specific achievements in retrospective radiation dosimetry 
have also resulted from the programme. The socioeconomic value of the programme can be evaluated in 
terms of life prolongation and improved quality of life through better medical care, or averted medical 
cost due to avoidance of adverse radiation effects. 
The analysis of the ecological consequences of the Chernobyl accident has thrown into relief which 
remedial actions are beneficial and which are less so. A new approach to radioecological studies has led 
to the development of models which may contribute to a comprehensive environmental strategy for 
response to any future accident. Finally, much has been achieved in developing and disseminating within 
the EU computer programmes for risk assessment and decision support following a nuclear emergency. 
163 
17. 
The transition from FP3 to the Fourth Framework Programme (FP4), with a smaller number of well-
targeted research areas, increased numbers of partners in each project, increased multi-national 
cooperation, increased participation by industry and by small and medium-sized enterprises has led to 
improved management of what was already an effective and productive European research effort. In 
several areas of this programme, the work of European organisations represents the leading edge of world 
science, and these areas do represent real opportunities for added economic value to European industry. 
An important outcome of this EU-wide research and training programme is the maintaining of scientific 
and technical competence even in those EU member states where such knowledge is not provided as a by­
product of operating nuclear power plants. The continuing safety of populations can be assured only if 
sufficient knowledge is available in all countries. This may require that a new generation of suitably 
qualified scientists emerges to replace those who are retiring. Active participation in future well-targeted 
EU research programmes is an excellent means for fostering a new generation of specialists. 
The common scientific base, to which this research programme has made significant contributions, 
allows EU member states to make their own policy decisions, but helps to ensure that the logic by which 
those decisions evolve is apparent. 
The major method of communication of results obtained in this programme is via publication in refereed 
scientific journals, but publications by the Commission have made a useful contribution in dissemination of 
information about the programme. European and wider international scientific audiences have been 
reached successfully by the numerous symposia, workshops and conferences organised within the 
programme. However, the use of the results of this programme by other Directorates of the Commisson, 
e.g. in setting standards, in improving public confidence, and in assistance to countries in central and 
eastern Europe, has been less than we would have hoped. 
The development of FP4 represents a significant increase in detailed specification by the Commission 
of the types of studies which would be supported, although the programme evolved from wide discussions 
to allow input from individual scientists. It is important that a balance is maintained to ensure that 
opportunity exists for innovative scientific groups in member states to obtain EU funding for work in new 
or unorthodox areas where the chance of success may be low but the potential value of success is 
disproportionately high. 
FP4 provides an excellent base for defining needs for the Fifth Framework Programme (FP5). 
However, there will be advantage in reviewing objectives and time schedules in view of end-user needs. 
End-users of each of the programmes should be identified, and links established between the research 
teams and the users. Several results from the programme can be tailored so that they are used by small 
and medium-sized enterprises, e.g. in radiation control, reactor safety, waste management and emergency 
preparedness. 
By concentrating further research on risk relevant items, extensive knowledge about severe accidents 
should be available by the end of FP5, providing a common scientific basis of understanding at the 
European level about conceivable reactor accidents and corrective measures. This knowledge can be 
utilized in appropriate accident management procedures and, where applicable, design features. 
Underlying knowledge for waste repositories should improve up to the time of actual disposal. The 
main task is now to show that all the necessary studies and experimental investigations are carried out to 
achieve, to the required degree, the safe performance of the entire waste management system - especially 
the geological disposal. In this context, the long term tasks require further effort, not least because of the 
wide range of potential geological formations in the EU member states. 
We consider that no further research work will be required on decommissioning, since the current 
programme in FP4 will lead to demonstration of a comprehensive range of successful techniques on a 
wide range of reactor stations and other nuclear installations. 
164 
17. 
Study of human protection against radiation resulting from nuclear power activities requires continued 
research. This area is important in demonstrating improving quality of life for European populations, and 
relieving them of some of their fears. It is an area of scientific excellence and exceptionally broad 
European collaboration, where the activities of the Commission can guide and ensure enhanced output of 
national research. Even if some aspects of the programme in the radiation protection area are of no direct 
relevance to nuclear safety, they are of great significance to the protection of the patient in medicine and 
should be continued. 
We consider that the area 'Mastering Events of the Past' can be merged into the research areas 
mentioned above. 
Detailed recommendations at the end of the main text include suggestions for further improving the 
Commission's administrative procedures, so as to leave more time for the contractors to concentrate on 
their research. 
165 
17. 
II. COMMISSION'S RESPONSE 
The Commission acknowledges the work of the independent evaluation experts who were confronted with 
the task to evaluate for the first time the Euratom research and training areas on Reactor Safety, 
Decommissioning and Radioactive Waste Management and Disposal together with the Radiation 
Protection Research. 
Since the Radiation Protection Research contains some subject fields, such as radiation from natural 
sources and medical use of radionuclides and ionising radiation, which are not relevant for all the research 
areas of the SP Nuclear Fission Safety, the evaluation panel had the difficult task to situate the various 
research achievements of the past as well as to emphasise research objectives for the future. 
It is in line with the Commission's strategy for planning future research that a more coherent approach 
should be followed and that the protection of the population and protection of the environment will remain 
a main objective of the NFS research. A further important aim will be to stimulate the conservation of 
expertise and industrial activities which assure European competitivity in the nuclear field. Equally 
important, when defining new research objectives, a greater emphasis will have to be put on the end-user 
of the research results and on the establishment of 'closure criteria' in order to set more precise time limits 
to the research projects. 
The Commission concurs with the specific conclusions that research on severe reactor accidents should be 
continued and that innovative European reactor designs should be supported in the future. Furthermore the 
recommendation will be considered that safe performance of the waste management and in particular 
geological disposal will be of special importance for this programme. 
The Commission shares the panel's opinion that an EU programme of radiation protection research must 
continue to ensure safe application of ionizing radiation in the different fields, as far the public, the 
workers and the environment are concerned. It will thus help to establish an objective framework to 
prevent distorted public perception of radiation risk from denying to Member States the undoubted benefits 
of wider uses of radiation in medicine and industry, and not just in nuclear energy. 
The Commission underlines that one particular role of its research programme in NFS is to contribute to 
establish a baseline for apparent logic in the policy decision making of the EU Member States. The 
continuing European programme will help to assure complementarity and quality of work across the 
Member States. 
The Commission takes up the panel's view point that more attention will be paid to cooperation and 
management aspects, in order to improve the efficacy of the EC's effort. This includes that scientific 
excellence and management skills will remain the yardsticks for future selection of research priorities. 
The EC's research programme will also be conceived to attract young scientists to these fields of 
research where European competence risks to loose territory on the global scale. 
166 
17. 
III. EVALUATION PANEL 
Mr BENNETT, B.G 
UNSCEAR Vienna 
Secretary/Committee on the effects of atomic radiatons 
Mr ESCHRICH, H. 
retired 
Nuclear reactor specialist 
Mr FEATES, F. 
University of Manchester 
Professor/environmental radiation chemistry 
Mr GUILLAUMONT, R. 
CNRS 
Professor, Institut de Physique Nucléaire 
Mr MARCUS, F.R. 
NKS (Nordic Nuclear Safety Research) 
Expert in nuclear safety research 
Mr VAYSSIER, G. 
Nuclear Safety Consultancy 
Consultant 
Mr WAMBERSIE, A. 
Université Catholique Louvain (UCL) 
Professor, faculté de médecine, radiobiologie et radioprotection 
Mr BERRY, R.J. 
Consultant 
Nuclear Safety Consultant 
167 

18. 
1996 FIVE-YEAR ASSESSMENT 
CONTROLLED THERMONUCLEAR FUSION 
169 

I. EXECUTIVE SUMMARY 
Recommendations 
1. Fusion is one of the few energy sources which might make a significant contribution to satisfy the 
growing need for electricity from the middle of the 21st century onward. Taking into account 
intrinsic safety aspects, potential environmental advantages and the wide availability of fuel, it is 
important for Europe to have this option open. The long-term and the wide scope of fusion 
development justify, in application of the principle of subsidiarity, the continuation of the direct 
involvement of the European Union in this R&D area, and of the full integration of the efforts of the 
Member States (and Switzerland) in a European Fusion Programme. 
2. The Board confirms the validity of the long-term R&D strategy recommended by previous panels and 
endorsed in the 1994 Council Decision: on the path to the Demonstration reactor (DEMO), only one 
large device is needed i.e. a tokamak experimental reactor (ITER). 
3. The Board, welcoming the strong efforts developed by the Programme to focus on ITER most of the 
scientific, technological and industrial activities, recommends that this effort be continued and further 
steps be taken to increase the involvement of industry in the Programme. 
4. Fusion R&D has now reached a stage where it is scientifically and technically possible to proceed 
with the construction of the first experimental reactor, and this is the only realistic way forward. 
Starting the construction of ITER is therefore recommended as the first priority of the Community 
Fusion Programme under the Fifth Framework Programme. 
5. ITER should be built in Europe, as this would maintain Europe's position as world leader in fusion 
and would be of great advantage to European industry and laboratories. This would, however, 
require an average increase of at least 50% in the Community funding for Fusion in the first decade 
of the next century, accompanied by the phasing out of JET, a redirection of the Community funding 
of the Associations, and a substantial contribution from the host country. For the Fifth Framework 
Programme the additional financial requirement would be limited to the order of 200 MECU. 
6. If it turned out not to be possible to have ITER in Europe, the firm recommendation of the Board is 
to maintain a strong participation in ITER as the first priority of the European Fusion Programme. 
7. The Board recommends that, with regard to the construction of ITER : particular attention be 
devoted to the organisation of System Engineering and to the consequent optimisation of the 
industrial participation; the contributions by the partners be provided mainly in kind; and a substantial 
fraction of the ITER staff be seconded by the partners to the project for periods appropriate to their 
tasks, and then return to their parent institutions to optimise the circulation of know-how . 
Assuming this main strategy is implemented, and ITER is built, the following further recommendations 
apply : 
8. As soon as a firm decision on the construction of ITER is taken, the JET programme should be 
reassessed and any possible, further activities concentrated on key issues for ITER, within the 
stringent financial constraints which will be imposed by ITER construction. Particular attention 
should be devoted to the management of JET personnel, as the termination of the project 
approaches. 
9 Association programmes on both physics research and technological development, should be pursued 
with the aim of contributing to ITER and optimizing the database for DEMO. Physics research to be 
conducted on small and intermediate size devices in the Associations, should deal with plasma 
physics and concept improvement. Long-term technology work should include the development and 
characterisation of appropriate reactor-relevant materials; this will require the construction of an 
171 
intense 14MeV neutron source. The design and development of the DEMO blanket should continue, 
taking full advantage of the opportunity to test blanket modules in ITER. 
The Board supports the development of the Stellarator line, so that future strategic decisions on 
DEMO can be founded on a sufficiently broad physics basis. 
The Board recommends that no new initiative be launched on the line of Reversed Field Pinch 
devices as long as no new favourable data are available. 
New "clustering" initiatives between the Associations around specific themes, similar to those already 
successfully introduced, should be promoted. 
10. The Board noted the substantial strengthening over recent years of the interaction between the Fusion 
Programme and industry, centered mostly on ITER activities. Industry will have to play the primary 
role in the final development of fusion: it is therefore recommended to increase progressively its 
involvement, particularly in the System Engineering area. 
In addition, the following recommendations are made regarding other aspects of the Programme's 
activities: 
11. The Board was favourably impressed by the work performed on the assessment of safety and 
environmental aspects of fusion power and strongly encourages a renewed effort in this field. 
12. A substantial effort should be devoted to socio-economic research on fusion. It needs to be multi-
disciplinary and deal with issues such as: cost elements of fusion power, public awareness, 
democratic governance of complex systems, and value change. 
13. The watching brief on Inertial Confinement Fusion should be maintained, and in this frame a co­
ordination of the civilian national efforts in Europe is recommended. 
14. The high rate of human mobility achieved by the "Mobility Scheme" is essential to the cohesion of 
the programme and should be maintained. The domain of application should be increasingly open to 
industry. 
75. The decentralised management of the Programme is efficient, flexible and well accepted by the 
scientific, technological and industrial circles. With appropriate adaptations, it seems suitable for the 
difficult tasks ahead. The ITER project will, of course, call for a specific, central management of 
the European contributions when the project reaches the construction phase. 
And finally, external events may invalidate the basic assumptions of this Report : 
16. A breakdown of the ITER framework for any reason would result in the inability to go forward with 
the planning and construction of ITER on the lines broadly outlined in this report. In this event the 
Board recommends that there should be a complete reassessment of the European Fusion Programme 
in the light of this new situation. 
172 
II. COMMISSION'S RESPONSE 
1. At its meeting on 26 November 1996, the Consultative Committee for the Fusion Programme 
(CCFP) has unanimously adopted the preliminary1 views, herewith attached, on the Board 
Findings and Recommendations. The Commission, which is a formal Member of the CCFP, 
adheres fully to these views. 
2. Regarding the Board recommendations on ITER, it could be added that the recent expression of 
interest in hosting ITER from a EU Member State confirms, as pointed out by Mrs Cresson at the 
Research Council on 7 October 1996, the will of Europe to play a significant role in this 
enterprise. But the extent of this role cannot be defined before the financial involvement is fully 
clarified. Quadripartite explorations have just started among the ITER partners which should 
allow, among others, to draw up, in a year or so, realistic assumptions on the sharing of the 
financial burdens between the potential host and the other partners. 
Preliminary views of the CCFP on the findings and recommendations of the 1996 Fusion evaluation 
board 
• The CCFP welcomes the recognition by the Board of the success of R&D work performed and of the 
validity of the strategy followed by the European Fusion Programme. It accepts the need for enhancing 
industrial involvement, pursuing actively safety and environmental studies, and launching studies on 
socio-economic aspects. 
• The CCFP understands that arguments in support of the Findings and Recommendations will be found in 
the full report. 
• The CCFP wishes to highlight the following points from the Findings and Recommendations of the Board: 
Fusion is mature for ITER, and the whole momentum of the Programme would be lost if ITER were 
not built in the near future. 
Today, Europe has the leading role in fusion research. Now that the construction of the first fusion 
reactor approaches, with an increasing involvement of industry, it would be difficult to understand 
that Europe abandons its leading role: ITER should be built in Europe. 
Fusion has an important role to play in meeting the growing energy needs of the coming generations. 
The CCFP agrees with the Board in pointing out that fusion has to be seen in the context of a much 
wider energy research effort. 
Following the principle of subsidiarity, not all energy options have to be developed at Community 
level, but because of its long-term and wide scope, fusion necessarily does. 
• The CCFP fully realizes that important additional work will have to be performed as a consequence of the 
Board report: 
In preparation of the ITER site decision, a thorough description of the long-term impact on both the 
host and the non-host programmes, respectively, has to be produced. 
The future programme will have to be re-directed and re-focused after the decision on ITER siting 
and construction. 
' The "Findings and Recommendations" of the 1996 Fusion evaluation Board are based on an in-depth analysis, performed by the 
Board, which will be included in the full report yet to be finalised. 
173 
• The CCFP agrees with the Board that a negative decision on ITER would have such a large impact, that a 
complete re-assessment of the European Fusion Programme would become necessary. 
• The CCFP considers the Findings and Recommendations of the Board report to be an important input for 
the preparation of the Fifth Framework Programme. When the full report of the Board will be available, 
the CCFP will provide its Opinion. 
174 
/18. 
m. EVALUATION PANEL 
Prof. Sergio BARABASCHI 
Former Under-Secretary of State for Research and Technology, Italy, Chairman of the European 
Industrial Research Management Association (EIRMA) 
Dr Claus BERKE 
Former Executive Director at Siemens AG and former President of the European Nuclear Forum 
Dr Feliciano FUSTER JAUME 
Chairman and Chief Executive Officer of Empresa Nacional de Electricidad (ENDESA) and its Group of 
companies 
Sir John HILL 
Fellow of the Royal Society, former Chairman, United Kingdom Atomic Energy Authority 
Prof. Lars INGELSTAM 
Professor of Technology and Social Change, Linkôping University 
Prof. Francis TROYON 
Professor of Physics, Ecole Polytechnique Fédérale de Lausanne, Chairman of the JET Council 
Prof. Harry van der LAAN 
Professor of Astronomy, Univesity of Utrecht, former Director-General, European Southern Observatory 
Prof. Jean-Paul WATTEAU 
Former Rector-Chancellor of Académie de Grenoble 
175 

19. 
1996 FIVE-YEAR ASSESSMENT 
JOINT RESEARCH CENTER 
OVERALL EVALUATION AND SYNTHESIS OF 
VISITING GROUPS REPORTS 
by 
Professor Juan M. Rojo 
Universidad Complutense de Madrid 
SPAIN 
177 

19. 
L EXECUTIVE SUMMARY 
I have the clear impression that further progress has taken place in the JRC since the encouraging analysis 
of Sir Hermann Bondi three years ago. Both the management of the Centre and the Board of Governors 
guidance deserve much credit for this steady evolution. It is also worth remarking that most of the 
recommendations made in the previous evaluation have been actually implemented. I am confident that 
the recommendations that are included in the excellent reports elaborated by the present Visiting Groups 
will also be of great benefit to the Institutes. 
Under the guidelines of the Council, the JRC has obtained a much clearer definition of goals and has 
become much more open and accessible to the outside, with increasing activities of support to the 
Commission services and Union policies and widespread networking with research organisations, 
authorities and industries. In particular, the Centre has reacted quickly to the new mandate of 
competitiveness on all fronts: it has successfully engaged in shared cost actions of the Framework 
Programme of the Union and is becoming progressively more involved in joint ventures with industries. 
With the refreshment of the management team, new and intensive drives towards project control, quality 
management and marketing are being introduced to increase the visibility of the scientific and technical 
activities and to render the Centre's operation more transparent and understandable. Moreover, a change 
of culture is discernible in which much of the staff now seems convinced of the benefits accruing from the 
more competitive stance of the JRC. Continuous support of this succesful policy is imperative and, 
following the advice of some of the Visiting Groups, I recommend fostering it even further by setting up 
on a progressive basis a user advisory panel for each Institute, that would promote relevance in project 
missions and channel the needs of the users and their views on the operation of the Institutes, particularly 
in relation to transfer. 
In order to accomplish their mandated activities of institutional and competitive research the JRC has to be 
an exceptionally qualified partner and it is gratifying to note that the JRC has succeeded in maintaining a 
number of core areas of scientific excellence and is struggling to establish new ones. To nurture the search 
for quality, I propose the creation of a permanent scientific advisory panel reporting to the Board of 
Governors. This panel would assess the quality of the ongoing research, with particular regard to 
identifying high quality units in the Institutes, and would also advise on exploratory research. They would 
also provide valuable inputs to the Visiting Groups. 
An adversary of excellence is uncontrolled diversification. Focusing is imperative as it seems that the JRC 
has much too broad a spectrum of subjects. It is surely much better to have a restricted number of areas 
in which the JRC is recognised as first-class than to have at one's disposal a kind of low-profile general-
purpose partner. After all, one must consider JRC activities with reference to subsidiarity and there is no 
point in duplicating national facilities if there is no definite advantage. Much in the same line, the JRC has 
to make every effort to avoid routine work, even if it is requested by third parties. 
In order to maintain the positive evolution of the JRC, special attention should be given to the issues of 
staff management. In relation to recruitment, I have the most serious reservations for the future. As noted 
in previous evaluations one has to remark yet again on the difficulties in trying to use a system apparently 
designed to recruit government officials to obtain high grade scientists. It is far too rigid, and much too 
slow to meet the demands of 1996, let alone those of the coming century. Some slight improvement has 
been noted but a substantial overhaul is needed and I suggest the urgent setting up of a specific panel to 
deal with the subject including possible changes of current regulations. I have great hopes that the recent 
transformation of the JRC into an independent Directorate General might help to implement the specific 
requirements of a research centre in the EC structure. Mobility is another key issue that requires 
revitalisation. The continuous flux of ideas to the Institutes should be encouraged by stimulating the 
temporary attachment of trainee scientists from other institutions and the regular presence of research 
students, particularly at the post-doctoral level. 
179 
19. 
A final word about energy. I fully support the concerns of my distinguished predecessor regarding the 
decline in nuclear relevant research. Independently of personal or political attitudes towards nuclear 
energy in Member States, nuclear safety, waste treatment, non-proliferation and other issues, are still 
with us. They are no respecters of national frontiers and deserve our attention at European level. It is 
worth underlining that the JRC has first class expertise in some areas of this field. 
In summary, I believe that the JRC is now on the right track and that present policies are workable and 
satisfactory and, being supported by efficient management, have already transformed the Centre into a 
useful partner of institutions and industry in Europe. In order to maintain this satisfactory trend a number 
of steps, for which recommendations are given, should be taken, at least in relation to staff management, 
support of quality, and focusing of research subjects. 
1. Introduction 
Research, technological development and demonstration (RTD) is one of the key policies of the present 
Treaty of the Union, being third in rank in terms of economic volume. The reason for this high profile is 
the conviction that those basic objectives of the European Union (EU), industrial competitiveness and 
improvement of the quality of life, can only be built on the solid foundations of science and technology. 
Whereas the commitment of the EU to RTD has been permanent and increasing, the emphasis on the 
different areas, or even in the type of activities required, has been continuously changing. The role of the 
Joint Research Centre (JRC) has also undergone a number of realignments. 
Following the dwindling interest in fission research in the past, specialised nuclear establishments in 
Member States had to face arduous reconversions. The JRC was no exception and, in fact, some of its 
present specialisations, like environment or safety, coincide with the goals of some of the reconverted 
national nuclear centres. On these grounds it is not unexpected that in the past the JRC strategy was often 
defensive, nor is it surprising that in the last few years, there have been a number of definitions from the 
European decision-making bodies in relation to the goals to be attained by the Centre and to the various 
procedures of evaluation and assessment of its performance. The last of them, in 1994, is the so-called 
'modified role' of the JRC as agreed by the Council, Parliament and Commission. Among other issues this 
new role emphasised the need for the JRC to be more involved in the denominated 'competitive actions', 
often in partnership with enterprises. 
Although the present assessment is supposed to cover the period of five years 1992-1996 (with full 
recognition of the previous evaluation for the 1992-1994 period, the Bondi report), the introduction at 
mid-term of the modified role of the JRC referred to above suggests that stress should be laid on the 
performance of the Centre in terms of this modified role, i.e. during the last two years. This has been well 
understood by the Visiting^ Groups to the individual JRC institutes which, in their reports, have given 
special attention to the implementation of the new more competitive activities. These reports underline the 
positive evolution of the JRC performance over the last two years, confirming the earlier expectations of 
the Bondi assessment. 
The significance of the individual reports of the Visiting Groups can hardly be over-emphasised as they 
have been conducted in the most competent and professional way. Apart from a limited input derived 
from personal meetings with Institute Directors and staff, most of the present assessment is based on those 
reports. The scheme of my own report is as follows: In Section 2, I underline key points from the 
individual reports, trying to look for unifying issues. I then proceed to a more general discussion of the 
achievement of the JRC in the last five years (Section 3), keeping in mind the objectives defined in the 
corresponding decisions of the European bodies. In Section 4, on the basis of the positive evolution of the 
Centre, I look into the future in search of those R&D problems which the JRC can actually undertake 
more efficiently than national laboratories. Finally, Section 5 deals with conclusions and 
recommendations, the latter being general in scope. They are to be considered as complementary to the 
specific ones suggested for the individual Institutes by the Visiting Groups. 
180 
19. 
2. Analytical remarks on the Individual Institutes 
The full reports of the individual Visiting Groups are published separately. They encompass an evaluation 
of the activities of the Institutes, in the light of the objectives defined by the Council decisions, and 
specific recommendations for the future. In the present Section 2, I attempt to select from those reports 
some guidelines that I find most illuminating or representative of current issues. 
Institute for Reference Materials and Measurements (IRMM): 
* The Institute appears to occupy an important place in the EU structure of the sector. Indeed, it seems 
that if the Institute did not exist, a centre of this type would have to be created. 
* The Institute seems to have attained a satisfactory degree of competence and respectability, particularly 
in nuclear reference and measurements. It also embodies special experimental facilities. 
* Accountancy and management procedures require improvement and adaptation to the Institute's 
particular role. 
* Original research work must not be neglected. In this respect 10% of the resources seems a judicious 
figure for exploratory research. 
Institute for Transuranium Elements (ITE) : 
* In my view, this is the most qualified Institute in the JRC. Every effort should be made to maintain this 
status, particularly in recruitment issues in the face of coming retirements. 
* The Institute has also been successful in incorporating competitive activities into their former, mostly 
research oriented, ones. 
* Avoid remaining in areas which become routine: rather transfer work to other establishments when 
routine approaches. One should make sure that room for stimulation of creativity exists in all areas (i.e. in 
safeguards analysis). 
* In the two directions of exploratory research some novel ideas have originated recently in other 
European laboratories (i.e. the 'energy amplifier and waste burner'). ITE should evaluate them, report to 
the Commission and, if convenient, network. 
Institute for Advanced Materials (IAM) : 
* Very much restructured, the Institute has gained recognition in the field of high temperature materials, 
although there is still room for improvement in other fields. Diversification into already well-covered 
fields should be avoided. 
* The future of the tritium handling laboratory (ETHEL) should be examined in view of the evolution of 
the ITER and related fusion programmes. The relevance of this facility, in relation to existing ones in 
Member States, should be clarified. 
* Concern is expressed in a decrease in researchers being exchanged with external well-known 
institutions, compared to an earlier situation. This seems to me a critical issue. 
181 
19. 
* Transfer to the outside seems to be improving although research at the Institute should better remain 
generic and near, but upstream, of industrial research. The Visiting Group considers that shared cost 
actions should be favoured and encouraged. 
Institute for Systems, Informatics and Safety (ISIS) : 
* This Institute has been formed from the merging of two former ones, a new Director has been appointed 
and is, naturally, still defining goals and procedures. The management showed good perception as well as 
determination and should reap future rewards. 
* In recent times, the Institute has made a great effort in working together with external research and 
industrial bodies and this competitive attitude has to be stimulated and supported. The proposed exchanges 
of personnel with Directorates General in Brussels appear as a good practice for reinforcing a users 
approach with these units. 
* Quality seems to be generally satisfactory. Perhaps the very wide spectrum of interests of the Institute 
could be modulated by actual capabilities at the highest level. In some areas, improving scientific 
leadership seems desirable. 
* The Institute includes some unique facilities whose maintenance at the highest level of performance is 
mandatory. Enhanced use of these facilities by outside users is recognised. 
Environment Institute (EI) : 
* The Institute seems to be on the right track to the fulfilment of both institutional and competitive 
activities with a fair level of quality. The willingness of the management to convert it into 'the leading 
institute in Europe' with 'a high degree of visibility' has to be made compatible with increasing activities 
at national level in these areas and certainly requires careful focusing of the actions. 
* Along with a proper management control the management seems to be successful in stimulating the 
generation of new ideas, without which institutional and competitive activities could not be carried out 
efficiently. 
* Institutional activities are crucial in this domain. For example, the EI has produced amazing results 
from the comparison of pollution measurements in blind tests performed by different national reference 
laboratories with a scatter of more than 50%. It is hard to overemphasise the incidence of this work on 
implementation of environmental policies. 
* I strongly support the Visiting Group's suggestion of developing to capacity the activities in water 
research and management of water resources. An independent and neutral reference, along with the 
required scientific quality, is mandatory in this field. 
Space Applications Institute (SAI): 
* Institute with unique experience on télédétection and earth observation. It would benefit from stricter 
control of scientific quality (stimulate publications in prestigious journals) and performance in competitive 
projects (better definition of specific goals and milestones). 
* As remote sensing is generic by nature and aids in resolving so many different problems, one has to 
restrain the tendency of the Institute to become specialised in too wide a range of subjects. It would be 
better to restrict activities to the Institute's core competences (remote sensing and application of networks). 
* Interdisciplinarity requires improved co-operation with other Institutes, particularly the Environment 
Institute. 
182 
19. 
Institute for Prospective Technological Studies (IPTS): 
* Practically new-born, the Institute is starting to establish a sound reputation with its customers and set up 
infrastructure to meet their future demands. The location needs communications and world-wide-web to 
be developed to full strength. The Institute has still to demonstrate high quality and gain full recognition. 
* Previous recommendations on recruitment for Seville and launching of the European Science and 
Technology Observatory (ESTO) have been accomplished. 
* The general recommendations on 'focusing' are still more meaningful for IPTS. Work should not be 
encouraged in areas where European industries and laboratories have a long established tradition. 
* The Institute is taking advantage of its geographical position for developing leadership in Mediterranean 
and North African problems. I firmly believe that this approach should be reinforced, perhaps, as the 
Visiting Group indicated, by developing a programme outside the TEC (Technology, Employment, 
Competitiveness). 
3. General assessment of the JRC achievement 
In his report for the period 1992-1994, Sir Hermann Bondi stated that 'the JRC is well on the way to 
becoming a recognised centre of excellence in many of its areas of activity'. It is a pleasure to be able to 
confirm steady progress towards this desirable goal two years later. By taking together the five year 
period, from 1992 till the present, one can safely say that the JRC has successfully found a specific 
position in the European RTD system and that it is performing its role with increasing capabilities. 
Under the guidelines of the Council the JRC has attained a much clearer definition of goals and has 
become much more open and accessible to the outside, with increasing activities of support to the 
Commission services and Union policies, and in widespread networking with research organisations, 
authorities and industries. With the refreshment of the management team, new and intensive drives 
towards project control, quality management and marketing are being introduced to increase the visibility 
of the scientific and technical activities and to render the Centre's operation more transparent and 
understandable Moreover, a change of culture is discernible in which much of the staff now seem 
convinced of the benefits accruing from the more competitive stance of the JRC. It is perhaps too early to 
assess the impact of new distribution of tasks among the Institutes, as some merging and reorientation has 
recently taken place, but it is clear that further strengthenening the coordination among the Institutes is 
desirable. 
It is to be remarked that business-like management can never be at odds with scientific and technological 
quality. One cannot speak of transfer if there are neither significant ideas or know-how to be transferred. 
An unreasonable fixation on demonstrating 'usefulness' should not obscure the fact that in order to gain 
competitiveness, recognition and selected support of high quality units in the Institutes should be assigned 
a high priority. In fact, the JRC has succeeded in maintaining a number of core areas of scientific 
excellence and is struggling to establish new ones, although much thought has to be given to recruitment 
policy if one wants to maintain that promising trend. Also, the JRC seems to have been evolving into less, 
bureaucratic and more transparent managerial procedures. With regard to both items, it is satisfactory 
that the JRC has recently become an independent Directorate General as it is hard to use a system 
apparently designed to manage administrative offices to run an RTD centre. 
Whereas the present situation of the JRC can be regarded with confidence, there are a number of issues 
that are still a matter of concern. I refer to the most important ones perceived by the Visiting Groups, and 
in full agreement with my own experience: 
1. Personnel. A matter of common concern as this is certainly the key point for the future of the JRC. 
One should make sure that the massive retirement that is expected in the coming years is offset by hiring 
younger people of the highest quality. Recruitment procedures are still far from efficient. The whole 
183 
19. 
process of recruitment should be, tuned so that 3 years contracts could be used as a test of JttD capabilities 
for future candidates to temporary (semi-permanent) positions. I believe that only an overall evaluation of 
the problem can lead to substantial improvement and I specifically advise setting up a special panel for this 
purpose. As is often stated that much needed reforms are contrary to EU rules, maybe changing some of 
these rules for RTD centres could be one of the recommendations of that panel. 
2. Exchange of researchers. An action much praised by some of the Institutes in the past, particularly in 
strengthening capabilities and training young researchers in emerging areas. It also facilitates networking. 
However, it seems to be receding and correction of this trend should be undertaken. 
3. Focusing. There seems to be much too broad a spectrum of subjects. It is surely much better to have 
a restricted number of areas in which the JRC is recognised as first class than to have at one's disposal a 
kind of low-profile, general-purpose partner. 
4. Management and Accountancy. Some advance since the last report is recognised (i.e. rapid response to 
manpower needs in third party projects). However, improvements are still widely requested. An 
appraisal of general accountancy procedures is suggested. 
5. Co-ordination. Further advances since the last report are recognised, but stronger co-ordination among 
the Institutes is required. The Directorate of Programmes seems to be working at a very reduced staff 
level and requires examination. 
6. Nuclear energy. Apart from its historical origins, the JRC is the depository of much European 
knowledge in this subject, perhaps the only one in which it is recognised as the true European leader in 
some areas. In no way should this capital asset be wasted. 
4. A look into the future 
The JRC has received the mandate of performing three types of activity: institutional, competitive inside 
the Framework Programme (FP) and competitive outside the FP. Positive response in the period under 
examination to this challenge suggests that the time may be ripe for a further step, which would be that of 
progressively concentrating JRC actions on those in which its European added value (EAV) is fully 
exploited. In other terms it would mean proceeding from the question: Is there anything useful that we 
can do with the infrastructure and personnel in the JRC? to the much more engaging one: Are there 
R&D areas or activities in which the existence of a European-owned research laboratory is desirable or 
even necessary? Indeed, the overall RTD policies of the European Union (EU) are built on the principle 
of subsidiarity. As is well known, the EU Member States have a number of excellent laboratories, both at 
their universities or national institutes and at their industries. One may wonder whether there is in 
principle scope for an institute at European level, in which targeted research is performed. If the functions 
of such a centre can be recognised, one can further analyse if the JRC is presently capable of fulfilling 
these requirements. 
On the basis of the Council Conclusions of 26 April, 19941one could envisage three possible domains in 
which the JRC would have full EAV: 
The first domain would be that of specific and difficult problems that required, for their solution, critical 
mass of scientific and technical excellence. Excellence would be promoted by drawing from the pool of 
bright people in the different Member States and integrating that talent in first-class units of critical mass, 
that would furthermore pervade through the European RTD system by means of suitable networking and 
partnership. The second domain would be that of developing, maintaining and managing large 
experimental facilities, which could not be run independently by Member States. In the past a number of 
large experimental facilities have been set up, or upgraded, in Europe on the basis of transnational co­
operation among different countries. The third domain would be that of carrying out research and 
OJNo. C126, 7.5.1994, p.l 
184 
19. 
development actions in which impartiality and neutrality were mandatory. Needless to say, these virtues 
would have no practical use if they were not accompanied by scientific quality and recognition. 
Let us now turn to the crucial question, that is if one can realistically consider that the JRC can be 
competent in all these three domains at the present time: 
• First, its possible role as hosting units of excellence is analysed. Needless to say, I refer to both 
scientific and technical excellence. Gaining excellence is not at all an easy task: if an institute has 
little prestige, it will hardly attract the brightest people, which in turn will result in further 
stagnation. One has to recognise that the JRC has rarely become a reference point in the past, but 
the situation is steadily improving as discussed in Section 3. The present management has 
succeeded in injecting morale and determination into the staff, which now seems to believe in its 
own capabilities. Nevertheless, some bureaucratic remnants are still noticeable; for example, some 
officers when describing their work appear to be seized by the how but reluctant to emphasise the 
what. Full support of emerging new and promising ideas should be prevalent in all the Institutes. 
As support must always be selective, the management of the Institutes would much benefit from 
accurate information concerning the external appraisal of the quality of the different groups. I 
believe that the Board of Governors (and the Visiting Groups for their periodic assessment) would 
very much benefit from the creation of a scientific advisory panel that assessed the quality of the 
ongoing research, with particular regard to identifying high quality units in the Institutes, and 
advised also on exploratory research. One must not forget that only high quality units can actually 
provide EAV to their competitive partnerships with European industries and institutions. 
In the pursuit of excellence, there are a number of issues in which improvement is desirable. 
Ensuring that the best young people apply and that selection is efficient is second to none among 
the elements to maintaining high standards. My advice about personnel reforms is here crucial, as 
is my recommendation about focusing. A further suggestion in relation to mobility also seems 
relevant. It is always difficult to attract on a permanent basis first-class scientists or technologists 
who are well established in other institutions. However, incorporating them into the JRC groups 
for limited (but not too short) periods could result in immense benefit. Perhaps, previous practices 
of the Institutes to incorporate external researchers could be reinforced by developing a special 
programme which took advantage of sabbatical leave and included some convenient incentives for 
attracting these highly rated researchers. 
• Let me now turn to the second function, that of managing large facilities, which can become also 
reference points for networking with other European institutions. It is worth remarking that none 
of the recently developed large facilities in Europe has been undertaken under the JRC umbrella. 
Even the first European tokamak (JET), although funded by the Framework Programme of the 
EU, was conceived and developed outside the JRC. The positive evolution of the JRC, 
continuously alluded across this report, might change the attitudes of the Member States in the 
future and some thought should be given to alleviate some of the pending obstacles. The new 
Articles 130 k, 1 and n of the Union Treaty might also help to implement this function. 
• Finally, I address the function of impartiality and neutrality. It is often stated that Union policies 
require support from the JRC but very often this support is thought of in terms of routine or 
bureaucratic work. This is certainly wrong and creative and original work is mandatory for these 
purposes. This is perhaps the role in which the JRC has made a most visible progress recently. 
The support of this role should continue, but one must not forget that this support would become 
meaningless if the quality of JRC research failed and, consequently, a subtle equilibrium between 
the first and third domains is imperative. 
Last, but not least, the development of a body of independent knowledge in scientific and 
technological prospection is crucial for complementing the inputs from the different Member States 
in order to formulate the basis of future Framework Programmes. Previous Framework 
185 
19. 
Programmes have suffered from the lack of a neutral and recognised scientific and technological 
prospection and some of their programmes have become a mere list of Member States' interests, 
often without much EAV. The JRC has still to gain prestige and full recognition in this task but 
the rewards can be vast. 
5. Conclusions and Recommendations 
I have the clear impression that considerable further progress has taken place in the JRC since the 
encouraging analysis of Sir Hermann Bondi. Both the management of the Centre and the Board of 
Governors guidance deserves much credit for this steady evolution. In particular, it is worth remarking 
that most of the recommendations made in the previous evaluation have been actually implemented. 
The mandate to enter the competitive arena, and the general positive answer to this challenge in the 
different Institutes, has had a number of favourable consequences: 
0 Providing a check of quality (competing with national teams, having to attract potential users etc.) 
0 Opening the JRC groups to the outside, i.e. networking, etc. 
0 Improving management procedures, like quality management, project management and active 
marketing. 
The first recommendation is, consequently, that of maintaining and supporting the present policies. One 
has to realise, however, that even for consolidating the present advances, a number of improvements have 
to be made, particularly in the area of personnel management. Also, looking into the future, one has to 
aim at establishing the JRC as an institution with full EAV in the European RTD community and this will 
require further adjustments. In the following, I summarize a number of recommendations that derive from 
the analysis of the previous sections: 
a) Recommendations for specific action. 
• Set up urgently a special panel of experts, including specialists in European legislation, to 
advise on the subject of personnel status and recruitment procedures, including possible 
changes of current regulations. 
• Explore the possibilities of developing a special programme for the incorporation of first-class 
scientists into JRC laboratories during their sabbatical leave, that would reinforce previous 
individual action by the Institutes. Ensure a continuous incorporation of research students, 
particularly at post-doctorate level to the Institutes. 
• Set up an external scientific advisory panel, reporting to the Board of Governors, that would 
assess the quality of the ongoing research, with particular regard to identifying high quality 
units in the Institutes, and advise also on exploratory research. The panel would also help with 
the selection of new scientific staff and exchanges. 
• Set up user advisory panels at the Institutes in which the convenience of their existence has 
been manifested (for example, ISIS) and, on the basis of ongoing experience, extend them to all 
the Institutes. 
b) Recommendations for general policy 
• Avoid excessive proliferation of subjects in the Institutes. Instead, try focusing on activities 
leading to attainment of first-rate status in a limited number of subjects. 
• Emphasise, in the current accounts of the JRC activities, concrete results obtained in JRC 
Institutes rather than organisational issues. Periodically publish and disseminate news about 
those achievements. If possible select those cases in which the JRC has been given significant 
recognition (as opposed to 'listing customers'). 
186 
• Stimulate an ambience in which original, promising ideas can arise and support them in order 
to explore their possibilities Encourage publication in prestigious journals in order to gain 
recognition. Ensure that sufficient effort is devoted to exploratory research. 
• Avoid routine work even if it is sought by third parties. Try to step out of certain lines, and 
leave their continuation to other centres once the EAV of the action has vanished. Nevertheless, 
be careful in this respect in those areas in which the JRC has unique facilities. 
• Undertake the necessary steps to support the JRC as a candidate for future European large 
facilities. Explore the possibilities of using Articles 130 k, 1 and n as a 'driving force'. 
A note on the evaluation process itself 
In general, the evaluation process itself seems to fulfil its original purpose. In fact, previous evaluations 
have actually contributed to marked improvements in different aspects of JRC management. There are, 
nevertheless, aspects that can be ameliorated: 
• The Visiting Groups' task could be supported by an external evaluation of the quality of the 
output of the Institutes and by a unified view of the users appreciation of transfer, competitive 
partnership and related matters. The proposed panels could well fill this gap. 
• The first point of the terms of reference of the Visiting Groups states that the evaluation refers 
to all the objectives of the Council Decisions of 15 December, 1994 . However, the Visiting 
Groups' reports appear rather unbalanced in relation to the different objectives; for example, the 
sixth objective of the JRC "contributing to the reduction of scientific and technological disparities 
between Member States" does not seem to have played an easily recognisable role in the 
evaluation. 
Acknowledgements 
I acknowledge very useful discussions with the Board of Governors and members of the Management of 
the Centre and of the Visiting Groups. The exchange of views on the report with Sir Hermann Bondi has 
been of immense benefit to me. 
OJ No. L361, 31.12.94, p.l 14 and p.132 
187 
19. 
11^ MEMBERS OF THE VISITING GROUPS 
A. Institute for Reference Materials and Measurements (IRMM) - Geel 
Dr. Maria Grazia DEL MONTE (Chairwoman) 
Director 
National Centre for Reference Materials 
ITALY - ROME 
Mr. Peter J. JENKS 
Product Manager 
Biological, Elemental and 
Pharmaceutical Reference Materials 
Promochem Gmbh 
GERMANY - WESEL 
Mr. Robert KAARLS 
Director 
Nederlands Meetinstituut (NMi) 
THE NETHERLANDS - DELFT 
B. Institute for Transuranium Elements (ITE) - Karlsruhe 
Dr. Niels HANSEN(Chairman) 
Head 
Materials Department 
Riso National Laboratory 
DENMARK - ROSKILDE 
Mr. Richard HOOPER 
Director 
IAEA 
AUSTRIA - VIENNA 
Dr. Klaus JANBERG 
Geschaftsfuhrer 
GNS 
Gesellschaft fur Nuklear-Service GmbH 
GERMANY - ESSEN 
Mr. Jean van VLIET 
Manager 
MOX plant 
Belgonucléaire 
BELGIUM - DESSEL 
C. Institute for Advanced Materials (IAM) - Petten and Ispra 
Prof. Claude BATHIAS (Chairman) 
Conservatoire National des Arts 
et Métiers (CNAM) 
FRANCE - PARIS 
188 
19. 
Prof. Pier Giulio AVANZINI 
Direzione Area della Ricerca 
Consiglio Nazionale Ricerche 
ITALY - GENOA 
Mr. Staffan SÔDERBERG 
Director 
AB Sandvik Coromant 
SWEDEN - STOCKHOLM 
Prof. Dr. T. VARGA 
Leiter der Techn. Versuchsund Forschungsanstalt 
TU Wien 
AUSTRIA - WIEN 
D. Institute for Systems, Informatics and Safety (ISIS) - Ispra 
Mr. Peter BENTON (Chairman) 
Director General 
British Institute of Management 
UNITED KINGDOM - LONDON 
Prof. Jane GRIMSON 
Dean of Engineering and Systems Sciences 
Department of Computer Science 
Trinity College 
IRELAND - DUBLIN 
Mr. Theodossios P. TASSIOS 
National Technical University of Athens 
GREECE - ATHENS 
Prof. Dr. Dennis TSICHRITZIS 
Chairman of the Executive Board 
GMD-Forschungszentrum 
Informationstechnik GmbH 
GERNANY - SANKT AUGUSTIN 
Dr. F. YNDURAIN 
Director General 
CIEMAT 
SPAIN - MADRID 
E. Environment Institute (EI) - Ispra 
Prof. Carlos A. BORREGO (Chairman) 
Department of Environment and Planning 
University of Aveiro 
PORTUGAL - AVEIRO 
Prof. A.R. HARDY 
Research Director 
Central Science Laboratory 
UNITED KINGDOM - SLOUGH BERKS 
189 
19. 
Prof. Dr. A. KETTRUP 
Director 
GSF Neuherberg 
Institute of Ecological Chemistry 
GERMANY - NEUHERBERG 
F. Space Applications Institute (SAI) - Ispra 
Prof. Seppo URPO (Chairman) 
Director 
Metsâhovi Radio Research Station 
FINLAND - KYLMALA 
Prof. Dr. Herbert FISCHER 
Head 
Institut fur Météorologie und Klimaforschung 
Forschungszentrum Karlsruhe 
GERMANY - KARLSRUHE 
Mr. Roy GIBSON 
Former Director General of ESA 
FRANCE - MONTPELLIER 
G. Institute for Prospective Technological Studies (IPTS) - Seville 
Dr. Robert C. WHELAN (Chairman) 
Former Chief Executive of CEST 
UNITED KINGDOM - CAMBRIDGE 
Dr. Josef FRÔHLICH 
Head, Division of Systems Research 
Austrian Research Centre Seibersdorf 
Technology-Economy-Environment 
AUSTRIA - SEIBERSDORF 
Prof. Dr. Herbert PASCHEN 
Leiter des Instituts fur Angewandte Systemanalyse 
und Technikfolgenabschàtzung (ITAS) 
GERMANY - KARLSRUHE 
Prof. Walter ZEGVELD 
Chairman 
Rathenau Institute 
THE NETHERLANDS - THE HAGUE 
190 
Annex 
191 
ANNEX 1 
TERMS OF REFERENCE 
FOR THE 5-YEAR ASSESSMENT OF THE 
SPECIFIC PROGRAMMES 
1. Assessing relevance of programme objectives : 
Are the initial objectives still valid against evolving S&T, industrial and socio­
economic conditions ? 
Have the selection criteria as outlined in the Framework Programme Decisions 
been adhered to ? 
2. Assessing efficiency : 
Has the programme sought to achieve/achieved its objectives in a cost effective 
manner ? 
How efficient has the overall management of the programme been ? 
Have objectives been adopted against which performance can be evaluated ? 
Is improvement to the external monitoring process/procedure necessary ? 
3. Assessing effectiveness : 
Have the initial objectives of the programme been achieved, or, for the longer 
term strategy and objectives, is progress towards achieving them sufficient ? 
Have the results stemming from the programme been disseminated/exploited ? 
Have the overall objectives of the Framework Programme been achieved, 
particularly : 
• What have been the contributions to strengthening the scientific and 
technological bases ? 
• What has been the interaction with and/or what have been the contributions 
to relevant Community policies such as cohesion, competitiveness, agriculture, 
employment, cooperation for development, economic cooperation, fisheries, etc. ? 
• Is Community RTD activity sufficiently coordinated with that of the Member 
States ? 
How is European "added value "/cost benefit/subsidiarily best demonstrated ? 
Have the results stemming from the programme been disseminated/exploited ? 
4. Examples of major achievements : 
Outstanding S&T results and innovations. 
Major achievements as regard the contributions to other community policies. 
5. Lessons learned from programme implementation : 
What lessons have been learned from programme implementation ? 
Success stories of collaboration. 
Failure stories of collaboration. 
6. Recommendations for the next programme. 
The five-year assessment will pay particular attention to the coherence between 
Community andnational S&T policies with a view to enhancing their mutual consistency. 
192 
ANNEX 2 
TERMS OF REFERENCE 
FOR THE VISITING GROUPS FOR THE EVALUATION OF THE 
JOINT RESEARCH CENTRE'S INSTITUTES 
1. To evaluate progress in performing work in accordance with the objectives set out in Council Decisions 
94/918/EC and 94/919/Euratom, documents addressed to the Board of Governors, the JRC's Annual 
Work schedules, budget etc., taking into account the results of the last JRC evaluation performed in 
1993-94. 
2. To ensure that the Institute has effective mechanisms to set its local scientific objectives, monitor 
progress towards those objectives and evaluate outputs. 
3. To review the various research activities to assess: 
a) the relevance of the work with respect to Community needs, including the relevance of work 
performed for external customers, when applicable. In particular whether the objectives 
initially set for the Institute are still valid against evolving S&T, industrial and socio-economic 
conditions; 
b) the scientific quality and productivity of the Institute; 
c) whether full advantage is being taken of scientific opportunities in the light of funding and other 
constraints. 
4. To assess the effectiveness of the scientific management leadership within the Institute. 
5. To advice on whether the Institute has been making efficient, effective and economical use of resources 
in carrying out its programmes and management functions. Resources include manpower, money, 
services, facilities, data and equipment. 
6. To review the extent and effectiveness of the Institute's external scientific links, including cooperation 
with research organisations, the higher education sector, industry and government departments in the 
Member States. 
7. To examine the relationship of the Institute's work in general, to the mission of the Joint Research 
Centre and to its forward strategic planning as well as to the overall objectives of the Framework 
Programme. Therefore assessing : 
• the contribution towards the improvement of the competitiveness of European industry and the 
economic impact (both direct and indirect) that has already been plus that which is expected to 
be achieved; 
• The contributions to strengthening the scientific and technological bases; 
• The contributions to relevant Community policies such as environment, industry, agriculture, 
transport, employment, cohesion, etc. 
To assess the European added value of the Institute's activities on the basis of the subsidiarity 
principle. 
8. To make recommendations and report to the Board of Governors of the Joint Research Centre. 
193 


ISSN 0254-1475 
COM(97) 149 final 
DOCUMENTS 
EN 12 15 16 05 
Catalogue number : CB-CO-97-140-EN-C 
ISBN 92-78-18077-7 
Office for Official Publications of the European Communities 
L-2985 Luxembourg