Source: EURLEX
Language: en
Format: md

#### EUROPEAN COURT OF AUDITORS

# EN

###### Special Report No 8 2009

### ‘NETWORKS OF EXCELLENCE’ AND ‘INTEGRATED PROJECTS’ IN COMMUNITY RESEARCH POLICY: DID THEY ACHIEVE THEIR OBJECTIVES?

(pursuant to Article 248(4), second subparagraph, EC)

EUROPEAN COURT OF AUDITORS

2

EUROPEAN COURT OF AUDITORS

12, rue Alcide De Gasperi
1615 Luxembourg

LUXEMBOURG

Tel. +352 4398-45410

Fax +352 4398-46410

E-mail: euraud@eca.europa.eu
Internet: http://www.eca.europa.eu

###### Special Report No 8 2009

More information on the European Union is available on the Internet (http://europa.eu).

Cataloguing data can be found at the end of this publication.

Luxembourg: Publications Office of the European Union, 2009

ISBN 978-92-9207-408-1

doi:10.2865/63596

© European Communities, 2009

Reproduction is authorised provided the source is acknowledged.

_Printed in Belgium_

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

**Paragraph**

3
#### CONTENTS

**GLOSSARY**

**I–V** **EXECUTIVE SUMMARY**

**1–16** **INTRODUCTION**

**1–5** **OBJECTIVES AND THREATS FACING COMMUNITY RESEARCH AND TECHNOLOGICAL DEVELOPMENT (RTD)**

1–3 POLICY OBJECTIVES

4–5 STRUCTURAL WEAKNESSES AND AMBITIONS OF EUROPEAN RESEARCH

**6–16** **THE SIXTH FRAMEWORK PROGRAMME**

9–11 THE NEW INSTRUMENTS AND THEIR OBJECTIVES

12–16 PROJECT SELECTION AND MANAGEMENT

**17–18** **AUDIT SCOPE AND APPROACH**

**17** **AUDIT SCOPE**

**18** **AUDIT APPROACH**

**19–56** **AUDIT OBSERVATIONS**

**19–27** **CONTRIBUTION TO COMMUNITY RTD OBJECTIVES IN GENERAL**

22–24 EFFECTIVE RESEARCH COLLABORATION

25–27 PROJECT PLANS BETTER THAN IMPLEMENTED PROJECTS IN A NUMBER OF CASES

**28–43** **ACHIEVEMENT OF THE NEW INSTRUMENTS’ SPECIFIC OBJECTIVES**

29–35 NoEs’ INTEGRATION OFTEN REPLACED BY TRADITIONAL FORM OF COLLABORATION

36–38 IPS: LARGE PROJECTS, AND RELATIVELY LOW PARTICIPANTS’ FINANCIAL INVOLVEMENT

39–43 SME PARTICIPATION TARGET NOT FULLY ACHIEVED

**44–53** **THE COMMISSION’S MANAGEMENT OF NEW INSTRUMENTS**

45–46 UNCERTAINTIES ABOUT THE INSTRUMENTS’ SPECIFIC ROLE

47–49 MULTIPLE LEGAL PROVISIONS AND NOT ALWAYS SUFFICIENT GUIDANCE

50 THE NEGOTIATION PROCESS: MORE LENGTHY AND OFTEN NOT TRANSPARENT

51 FEWER AND LARGER PROJECTS BUT STILL A LARGE NUMBER OF PARTICIPANTS

52–53 WEAKNESSES IN PROJECT MONITORING

**54–56** **STIMULATION OF RTD INVESTMENT**

55–56 MORE EU FUNDS FOR RELATIVELY LESS PARTICIPANTS’ RTD INVESTMENT

**57–65** **CONCLUSIONS AND RECOMMENDATIONS**

**REPLY OF THE COMMISSION**

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

5
#### GLOSSARY

**European Research Area (ERA)** : A border-free zone for research, in which scientific resources
will be better deployed to create more jobs and to improve Europe’s competitiveness.

**RTD Framework Programmes (FPs)** : Multiannual programmes establishing, for a given
period, the scientific and technological activities to be funded by the Community, their
budget and the detailed rules for their technical and financial implementation.

**RTD Instruments** : Different types of projects defined by the FPs. Each instrument addresses
specific objectives and involves particular participation arrangements.

**Networks of Excellence (NoE)** : Instrument aimed at creating substantial and lasting integration of the research activities of the network partners, thus directly tackling the fragmentation of research activities in Europe in a given area.

**Integrated Projects (IPs)** : Instrument designed to achieve ambitious, clearly defined scientific and technological objectives of a European dimension.

**Gross domestic product (GDP)** : The total market value of all final goods and services produced in a country in a given year.

**Small and medium-sized enterprises (SMEs)** : Enterprises which employ fewer than 250 persons and which have an annual turnover not exceeding 50 million euro, and/or an annual
balance sheet total not exceeding 43 million euro.

**SMART objectives** : Specific, measurable, achievable, relevant and timed objectives, as
defined by the EU Financial Regulation.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

(iii) The Commission’s management revealed
a number of weaknesses, in particular at
the beginning of the implementation of
FP6, which affected the effectiveness of
the actions, in particular the uncertainty
about the new instruments’ specific role,
an insufficient guidance and weaknesses
in project monitoring.

(iv) As compared to FP5, FP6 did not succeed
in generating a significant increase in
terms of participants’ RTD investment.
The participation of the private sector
even diminished in relative terms. The
target of 15 % of the total budget of FP6
thematic priorities for SME participation
was not achieved.

**IV.**

The Court recalls its previous recommendation that spending programmes should be
based upon an explicit intervention logic,
linking the instruments to realistic objectives. In this context, the possibility of setting one single objective for each instrument
should be considered in order to ensure clarity of the instruments’ specific role. For each
programme there should be appropriate performance indicators to monitor the expected
outputs, outcomes and impacts.

7

#### REPLY OF THE EXECUTIVE COMMISSIONSUMMARY

**V.**

The Court also recommends that:

(i) The Commission should consider whether
NoEs and IP-like collaborative projects
have realistic objectives and provide significant advantages as compared with
traditional instruments for research collaboration. Concerning NoEs financed
under FP6, the Commission should
assess on a case-by-case basis whether
past achievements, potential EU added
value and prospects of self-sustainability
justify further funding under FP7.

(ii) The Commission should examine the
various reasons underlying the relatively
low level of participants’ RTD investment
compared to the goals pursued and propose specific measures. Appropriate data
should be made available to monitor the
catalytic effect of EU RTD funding. The
realism of expected targets, in particular
for SMEs and the private sector, should
be reassessed.

(iii) In view of improving in particular the
manageability of projects, their adequate
implementation and appropriate evaluation, the Commission should ensure
clear and timely guidance, a speedier
contracting process and better project
monitoring.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

8
#### INTRODUCTION

**OBJECTIVES AND THREATS FACING COMMUNITY**
**RESEARCH AND TECHNOLOGICAL DEVELOPMENT (RTD)**

**POLICY OBJECTIVES**

**1.** Research and technological development (RTD) has a key influence on
scientific and technological progress and innovation. Research is an
important priority for the European Union and its Member States so
that the high living standards of its citizens can be maintained and
improved.

**2.** The overall objectives of the European Community RTD policy [1] are to
strengthen the scientific and technological bases of Community industry and encourage it to become more competitive at international
level. Undertakings, research centres and universities are encouraged
to cooperate with one another in high-quality RTD activities to exploit
the internal market potential to the full.

1 Treaty establishing the European

Community, Title XVIII (Research

and Technological Development),

Articles 163 to 173. The activities are

so defined:

(a) implementation of research,

technological development and

demonstration programmes, by

promoting cooperation with and

between undertakings, research

centres and universities;

(b) promotion of cooperation in the

field of Community research,

technological development and

demonstration with third countries

and international organisations;

(c) dissemination and optimisation

of the results of activities in

Community research, technological

development and demonstration;

(d) stimulation of the training and

mobility of researchers in the

Community.

**AVERAGE ANNUAL RTD EXPENDITURE FROM THE COMUNITY BUDGET (FP4 TO FP7)**

8000

7000

6000

5000

4000

3000

2000

1000

0

|Col1|Col2|7 217|Col4|Col5|Col6|Col7|Col8|
|---|---|---|---|---|---|---|---|
||**2 761**<br>**3 425**<br>**4 061**<br>**4 447**|||||||
|||||||||
|||**4 061**<br>**4 447**|**4 061**<br>**4 447**|**4 061**<br>**4 447**|**4 061**<br>**4 447**|**4 061**<br>**4 447**||
|||**2 761**<br>**3 425**|**2 761**<br>**3 425**|**2 761**<br>**3 425**|**2 761**<br>**3 425**|**2 761**<br>**3 425**||
|||||||||
|||||||||
|||||||||
|||||||||
|||||||||

FP4 — EU-12 FP5 — EU-15 FP6 — EU-15 FP6 — EU-25 FP7 — EU-27

(1995 to 1998) (1999 to 2002) (2003 to 2006) (2004 to 2006) (2007 to 2013)

_Source_ : Legislative decisions for FP4 to FP7 (excludes Euratom).

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

9

**3.** Since 1984, European Community RTD activities have been implemented
through multiannual Framework Programmes (FPs) which define, for
a given period, the scientific and technological objectives, the broad
lines of the activities, the overall budget and its breakdown by action,
and the detailed rules for Community financial participation. Funding
in successive EU Framework Programmes has increased over time.
FPs are the largest single public source of research funding in the
European Union, with an overall share between 4 % and 5 % of total
public RTD funding.

**STRUCTURAL WEAKNESSES AND AMBITIONS**

**OF EUROPEAN RESEARCH**

**4.** European research is widely held to suffer from structural weaknesses.
The Commission has highlighted that [2] :

(a) there is insufficient and dispersed investment in RTD;

(b) insufficient human resources are devoted to research;

(c) there is a limited capacity to translate scientific breakthroughs
into innovative and competitive products and services; and

(d) research policies in Europe are fragmented.

**5.** The Commission and the European Council considered that realising
an ‘European Research Area’ (ERA) is a key step to overcome these
structural weaknesses [3] . The ERA would constitute an internal market
for research and technology, as well as a space for a better coordination of national and regional research activities and policies. The
European Council has set for the Union the ambitious goal ‘to become
the most competitive and dynamic knowledge-based economy in the
world, capable of sustainable economic growth with more and better
jobs and greater social cohesion’ [4] . In this perspective, the European
research effort should be increased, with the aim of approaching
3 % of gross domestic product by 2010 (up from 1,9 % in 2000). Two
thirds of this new investment should come from the private sector [5] .

2 See the Commission proposal

for Council decisions concerning

the specific programmes

implementing the Sixth Framework

Programme, COM(2002) 43 final of

30.1. 2002, Section 5.1 ‘Need for

Community intervention’, p. 71.

3 The creation of a European

Research Area (ERA) was proposed

by the European Commission

in its communication ‘Towards

a European Research Area’ of

January 2000. The objective of

creating ERA was endorsed by

the EU shortly afterwards at the

March 2000 Lisbon European

Council.

4 See the Conclusions of the

European Council of Lisbon,

March 2000, point 5.

5 See the Conclusions of the

European Council of Barcelona,

March 2002, point 47.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

10

**THE SIXTH FRAMEWORK PROGRAMME**

**6.** The European Community Sixth Framework Programme (FP6) [6] was an
instrument intended to assist in achieving the ambitious objectives
mentioned earlier. Its aim was to fund activities contributing both to
innovation and to the creation of the ERA, by fostering more integration and coordination in Europe’s fragmented research sector. FP6
covered, for the period 2002 to 2006, the full spectrum from basic to
applied research, the development of scientific and technical excellence and the coordination of European research. The programme
was structured around three headings:

—
integrating and strengthening the European Research Area;

—
structuring the European Research Area; and

—
strengthening the foundations of the European Research Area.

**FP6 MAIN COMPONENTS AND BASIC PRINCIPLES**

6 Decision No 1513/2002/EC of

the European Parliament and

of the Council of 27 June 2002

concerning the sixth framework

programme of the European

Community for research,

technological development

and demonstration activities,

contributing to the creation of

the European Research Area and

to innovation (2002 to 2006)

(OJ L 232, 29.8.2002, p. 1).

_Source_ : Commission’s guide ‘Participating in European Research’, 2nd Edition, February 2004

(http://ec.europa.eu/research/fp6/pdf/how-to-participate_en.pdf).

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

11

**7.** With a budget of around 17 billion euro for the years 2002–06, FP6
financed until the end of 2008 ‘indirect’ RTD activities worth 25 billion euro in estimated investments [7] . Most expenditure (around 75 %)
was carried out under the heading ‘Integrating and strengthening
the European Research Area’, within seven ‘thematic priorities’ [8], i.e.
programmes covering specific themes of research.

**8.** In view of developing their technological capacity and facilitating their
access to high-quality research, FP6 put special emphasis on small
and medium-sized enterprises (SMEs). An amount of 1,8 billion euro
was earmarked for this purpose, i.e. 15 % of the total budget of the
thematic priorities (one third more than in the previous Framework
Programme).

**THE NEW INSTRUMENTS AND THEIR OBJECTIVES**

**9.** In addition to the traditional instruments [9], two new instruments were
introduced by FP6 aiming especially at ‘Integrating and strengthening the European Research Area’:

—
‘Networks of Excellence’ (NoEs) were aimed primarily at creating
a substantial and lasting integration of the research activities of
the network partners, thus directly tackling the fragmentation of
research activities in Europe in a given thematic priority.

—
‘Integrated Projects’ (IPs) were designed to generate the knowledge required to implement the thematic priorities, by achieving
ambitious, clearly defined scientific and technological objectives
of a European dimension.

**10.** NoEs and IPs could be carried out in any of the seven ‘thematic priorities’
(see footnote 8). These instruments were recognised by the legislator
as being the main means to generate added value over and above that
which could be achieved through national efforts (European added
value), by integrating European research capacities. This was supposed to achieve a critical mass in terms of expertise, activities and
resources (staff, skills, competences, finances, infrastructure, equipment). Also the new instruments were supposed to play an important
role in achieving the target of spending 15 % of the budget of the
thematic priorities on SME (see paragraph 8).

7 ‘Indirect’ RTD activities

designate actions to be

implemented by grant contractors,

as opposed to ‘direct’ activities

carried out by the Commission

through its Joint Research Centre.

8 The seven ‘thematic priorities’

are: life sciences, genomics

and biotechnology for health;

information society technologies;

nanotechnologies and

nanosciences, knowledge-based

multifunctional materials, and new

production processes and devices;

aeronautics and space; food

quality and safety; sustainable

development, global change

and ecosystems; citizens and

governance in a knowledge-based

society.

9 Specific targeted projects,

coordinated actions, specific

support projects, Marie Curie

actions, integrated infrastructure

initiatives and Article 169 actions.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

12

**11.** Boxes 1 and 2 provide examples of the type of activities carried out in
‘NoEs’ and ‘IPs’.

**PROJECT SELECTION AND MANAGEMENT**

**12.** Within the Commission, FP6 is a joint responsibility of several Directorates-General (DGs) [10] . As in previous Framework Programmes, the
Commission published periodically calls for proposals describing the
broad areas of activity. Request for funding submitted by potential
beneficiaries were assessed with the help of independent experts
(‘evaluators’). Proposals were evaluated against the scientific and
technical criteria defined in the call for proposals, including management criteria linked with the proposers’ potential capacity to implement the projects. Projects considered most deserving of EU funding
were selected by the Commission and grants awarded.

**13.** Project progress was monitored and periodically assessed by the Commission, including by reference to the objectives specifically assigned to
each instrument. This monitoring was carried out with the assistance
of independent experts (reviewers) [11] .

10 Together with DG Research,

which is the main Directorate

General managing and

coordinating FP6 activities,

four other DGs also managed

parts of specific programmes:

DG Information Society and

Media, DG Energy and Transport,

DG Enterprise and Industry,

DG Maritime Affairs and Fisheries.

In addition, the DG ‘Joint Research

Centre’ is in charge of those

activities which are directly

implemented by the Commission.

11 See paragraph 25 and

footnote 19.

**WHAT SORT OF ACTIVITIES WERE CARRIED OUT IN NoEs?**

Every NoE is different. Examples of the specific activities carried out include:

Integration of activities implemented under the NoEs examined included the exchange of PhD students
and research staff, the organisation of joint workshops, the creation of virtual Internet-based research
working groups, the establishment of virtual labs and institutes, and the organisation of visits intended
to provide access to off-site facilities.

Joint research activities implemented under the NoEs examined include the funding of individual research
projects, often through internal calls for proposals implemented under the responsibility of scientific
task leaders.

Spreading of excellence activities included visits to industry, participation in international conferences,
organisation of summer school sessions and other training events, the dissemination of project information through a dedicated web page, newsletters, press releases, and the publication of articles.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

13

**14.** Projects undertaken under NoEs and IPs attracted almost 50 % of FP6
funds. According to the Commission’s data under the Seventh Framework Programme (2007–13) NoEs and those collaborative projects
that assume IP characteristics received 16 % of the total FP7 funding
allocated up to February 2009. [12]

**15.** FP6 financed 167 projects as NoEs, involving some 5 000 participants
and a cumulative EU contribution of 1,2 billion euro (against eligible
investment costs of 1,9 billion euro). On average, each NoE involved
around 30 partners and an EU contribution of 7 million euro per contract. Almost 700 projects were financed as ‘IPs’. They involved some
17 000 participants and a cumulative EU contribution of 6,5 billion
euro (against eligible investment costs of 10,7 billion euro). On average, each IP involved around 25 partners and an EU contribution of
9,5 million euro per contract.

**16.** Overall, FP6 available funds permitted the financing of only one out of
five proposals received. This ratio is similar to that of the US National
Science Foundation funding schemes, but lower than that of the previous Framework Programme (26 %). A higher proportion of ‘NoE’
proposals were financed compared to ‘IP’ projects.

12 Decision No 1982/2006/

EC of the European Parliament

and of the Council of

18 December 2006 concerning the

seventh framework programme

of the European Community

for research, technological

development and demonstration

activities, (2007 to 2013)

(OJ L 412, 30.12.2006, p. 1).

Under FP7, ‘Collaborative

Projects’ may include the

activities previously carried out

by ‘Integrated Projects’.

**WHAT SORT OF ACTIVITIES WERE CARRIED OUT IN IPs?**

The activities covered by a project carried under IPs fall within one of the following categories:

 - research and technological development activities directly aimed at creating new knowledge, includ
ing innovation-related and dissemination activities;

 - demonstration activities designed to prove the viability of new technologies that offer a potential

economic advantage but which cannot be commercialised directly (e.g. testing of prototypes);

 - training activities intended to contribute to the professional development of researchers and other

key staff, research managers, industrial executives (in particular for SMEs), and potential users of
the knowledge generated by the project.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

14
#### AUDIT SCOPE AND APPROACH

**AUDIT SCOPE**

**17.** The objectives of the audit were to assess to what extent:

—
Networks of Excellence and Integrated Projects contributed to
achieve the research policy objectives set by the Treaty as well
as their own specific objectives.

—
The Commission effectively supported the projects’ implementation, by providing the necessary guidance to beneficiaries, managing the contracts and monitoring the progress achieved; and

— FP6 had stimulated RTD investment.

**AUDIT APPROACH**

**18.** Audit evidence was collected through:

—
Review of documentation on FP6 (preparatory documents, legal
framework and Commission guidance, implementation data).

—
Visits to 36 project coordinators and participants involved in
14 projects in 15 different Member States [13] .

— A series of ‘round table’ discussions with 60 researchers from
44 different organisations participating in FP6 [14] .

—
A survey of 387 RTD organisations (274 programme participants,
104 proposers and nine non-participants).

—
An analysis of selected management areas playing a key role in
the performance of the instruments (including the review of the
implementation of eight calls for proposals and an analysis of
the conclusions of 399 independent project reviews conducted
before 31 December 2007).

—
The review of secondary evidence on the effectiveness of the
instruments.

—
Consideration of the opinions expressed by some associations
active in Community research.

13 Austria, Belgium, Denmark,

Estonia, Finland, France, Germany,

Italy, the Netherlands, Poland,

Portugal, the Slovak Republic,

Spain, Sweden and the United

Kingdom.

14 Round tables were organised

with the assistance of the

FP6 National Contact Point

coordinators in Finland, France,

Poland, Spain and the United

Kingdom.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

15
#### AUDIT OBSERVATIONS

**CONTRIBUTION TO COMMUNITY RTD OBJECTIVES**
**IN GENERAL**

**19.** According to the EC Treaty, Community RTD policy has the overall aim
to strengthen the scientific and technological bases of Community
industry and to encourage it to become more competitive at international level.

**20.** In the framework of a previous audit in the field of research, the Court
pointed out the lack of an explicit intervention logic, explaining how
the different instruments and programmes were supposed to contribute to Community RTD goals [15] . In addition, no SMART objectives [16]
and performance indicators were defined for the individual research
programmes, thus undermining the basis for sound monitoring and
evaluation [17] .

**21.** Bearing in mind the lack of an explicit intervention logic [18], the Court has
examined the contribution of the new instruments to the Community
RTD policy in general as regards the following aspects:

—
The effectiveness of the research collaboration put in place;
and

—
The scientific quality of individual projects as assessed by the
experts.

17 See paragraphs 28 and 30 of Special Report No 9/2007.

15 Special Report No 9/2007

concerning ‘Evaluating the EU

Research and Technological

Development (RTD) framework

programmes — Could the

Commission’s approach be

improved?’ (see paragraphs 22

to 37) (OJ C 26, 30.1.2008).

Intervention logic is defined by

the Commission as ‘the conceptual

link from an intervention’s

inputs to the production of its

outputs and, subsequently, to its

impacts on society in terms of

results and outcomes’ (European

Commission, ‘Evaluating EU

activities: A practical guide for the

Commission services’, July 2004,

pp. 87 and 106).

16 SMART stands for specific,

measurable, achievable, relevant

and timed. The definition of

SMART objectives constitutes

a general principle of the EU

Financial Regulation to ensure

sound financial management (see

Article 27(3) of Council Regulation

(EC, Euratom) No 1605/2002

(OJ L 248, 16.9.2002, p. 1)).

18 For example, the impact of the absence of an explicit intervention logic was already highlighted in 2004 by the panel in charge of

the Five-year Assessment 1999–2003, which stated that ‘…panels like ours are asked to fill a gap between, on the one side, evidence

mainly collected at project level and, on the other side, the higher level socio-economic goals of research policy. However, at the

moment the link is difficult to make due to the way the FP is planned. It lacks an explicit logic connecting the highest objectives

to the specific research and knowledge goals’ ‘Five-year Assessment of the European Union Research Framework Programmes,

1993–2003’, Report of the Independent Expert Panel chaired by Erkki Ormala (15 December 2004): Section 6 ‘Evaluating the

Framework Programme’, p. 19.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

16

**EFFECTIVE RESEARCH COLLABORATION**

**22.** The effectiveness of research collaboration was assessed by reference to
the actual achievement of project deliverables, the transfer of existing and new scientific knowledge as well as the use of good practices
concerning the way in which research is carried out.

**23.** The Court concludes that the FP6 instruments covered by the audit had
promoted a good level of research collaboration between project
participants. Indeed:

—
In the Court’s view all the projects visited were successful in
promoting international research collaboration between organisations from different sectors and disciplines.

—
The assessment made by independent experts shows that the collaboration between project participants was effective in a large
majority of cases (see _**Figure 3**_ ).

**REVIEWERS’ ASSESSMENT OF RESEARCH COLLABORATION ACHIEVED**

**REVIEWERS’ ASSESSMENT OF THE EFFECTIVENESS OF COLLABORATION**

_**Has the collaboration between the participants been eff ective?**_

**NoEs** **IPs**

101 projects 298 projects

1 %

1 %

**Yes**

**Partially**

**No**

67 %

_Source_ : European Court of Auditors — Horizontal analysis of project review reports as at 31 December 2007.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

76 %

17

**24.** Moreover, the survey of participants reveals an overall satisfaction with
the two instruments audited (see _**Figure 4**_ ). NoEs were found to be
particularly successful by public research centres and universities,
while private partners (including SMEs) considered collaboration less
fruitful. Concerning IPs, the cooperation was considered effective by
both public and private partners.

**PARTICIPANTS’ PERCEPTION OF THE PROJECT’S EFFECTIVENESS**

_**How satisfactory do you consider project in terms of …**_

**NoE** **IP**

_**Coordinators**_ _Satisfactory_ _Unsatisfactory_ _Satisfactory_ _Unsatisfactory_

... the promotion of high-quality research collaboration
100 % 0 % 98 % 2 %
between project participants

... the networking of participants 100 % 0 % 97 % 3 %

... coordinating the strategic planning of the participants’
86 % 14 % 89 % 11 %
research activities

... transfer of knowledge between participants 100 % 0 % 97 % 3 %

... the promotion of the mobility of research staff between 89 % 11 % 80 % 20 %
participants

_**Participants**_ _Satisfactory_ _Unsatisfactory_ _Satisfactory_ _Unsatisfactory_

... the promotion of high-quality research collaboration
92 % 8 % 92 % 8 %
between project participants

... the networking of participants 94 % 6 % 98 % 2 %

... coordinating the strategic planning of the participants’
83 % 17 % 80 % 20 %
research activities

... transfer of knowledge between participants 97 % 3 % 88 % 12 %

... the promotion of the mobility of research staff between 89 % 11 % 64 % 36 %
participants

_Source_ : Survey conducted by the European Court of Auditors.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

18

**PROJECT PLANS BETTER THAN IMPLEMENTED PROJECTS**

**IN A NUMBER OF CASES**

**25.** Project quality is associated with the concept of scientific and technological excellence and the degree of innovation. It is a normal
practice for research projects that their quality be assessed by peers.
For FP6, independent experts were involved in the initial selection
of proposals as well as in the monitoring and evaluation of projects
implemented [19] .

**26.** Project reviewers’ _ex post_ assessment is based on:

—
the progress towards the achievement of its objectives;

—
the adequate use of resources;

—
the degree to which research collaboration has been effective;

—
the soundness of project management, and

—
the use and dissemination of knowledge arising from the project.

19 ‘Evaluators’ participate in the

evaluation process, assessing the

merits of each of the proposals

received against the criteria

defined by the Commission in the

corresponding call for proposals.

Based on the evaluators’

assessment, the Commission

decides which proposals deserve

to be funded. ‘Reviewers’ intervene

well after the project has actually

started. They periodically assess

progress achieved by a given

project during a reference period,

on the basis of an assessment

form prepared by the Commission.

Based on this assessment, the

Commission decides whether

the project may continue or if,

on the contrary, it should be

terminated before the planned

completion date. ‘Evaluators’ and

‘Reviewers’ are selected from the

same database constituted by the

Commission at the start of the

programme.

**REVIEWERS’ ASSESSMENT OF PROJECT QUALITY**

**OVERALL ASSESSMENT OF THE PROJECT’S QUALITY**

**NoEs** **IPs**

101 projects 298 projects

0 %

**Good to excellent project**

2 %

**Acceptable project**

**Unsatisfactory project**

**No reply**

36 %

_Source_ : European Court of Auditors — Horizontal analysis of project review reports as at 31 December 2007.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

19

**27.** The Court’s analysis of the experts’ review reports shows that more
than one third of projects were qualified as ‘good to excellent’ (see
_**Figure 5**_ ). Only six out of 399 projects implemented as NoEs and IPs
were considered unsatisfactory. However, a large number of projects
(174 projects, or 44 % of all projects reviewed) were assessed as not
more than ‘acceptable’. Considering that experts had qualified all
projects as ‘excellent’ at the time of their selection by the Commission, the review results indicate that project quality is declining in
the implementation phase.

**ACHIEVEMENT OF THE NEW INSTRUMENTS’**
**SPECIFIC OBJECTIVES**

**28.** As previously indicated (see paragraphs 9 and 10), NoEs were aimed primarily at creating a substantial and lasting integration of the research
activities of the network partners. IPs were designed to generate the
knowledge required to implement the thematic priorities, by achieving ambitious, clearly defined scientific and technological objectives
of a European dimension. For both instruments, the emphasis was on
integrating a ‘critical mass’ of expertise, activities and resources as
a pre-condition for long-term research activities and partnerships as
well as for achieving a real impact in scientific, industrial or economic
terms. The specific target for SMEs has to be considered in view of
their crucial role in the European economy.

**NOES’ INTEGRATION OFTEN REPLACED BY TRADITIONAL**

**FORM OF COLLABORATION**

**29.** In view of assessing the extent to which NoEs had achieved their specific
objective of promoting durable integration of the research activities
of the network partners, the Court analysed whether:

—
The resources put into the network could be considered significant
in relation to each participant’s overall budget.

—
NoEs had significant control on the deployment of the resources
made available for the project.

—
The high-level researchers initially envisaged had actually been
involved in the project.

—
NoEs had progressed towards long-term research activities and
partnerships beyond the duration of Community funding.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

20

**30.** The Court considers that, notwithstanding their success in promoting research collaboration and projects of reasonable quality, NoEs
audited have not achieved their specific objective. In particular:

(a) Most participants allocated a relatively small proportion of their
research capacities to the network.

(b) NoEs failed to reach control over resources made available and to
ensure the adequate coordination of project activities. Participant
organisations did not accept that NoEs’ governance structures
decide how networked resources should be used.

(c) In most cases the involvement of key high-level scientists in NoEs
was not realised.

(d) Self-sustainable long-term research activities and partnerships
were not achieved for any of the audited NoEs, thus making future
collaboration subject to continued public support.

**31.** The results of the project reviews carried out by the independent experts
show that full restructuring of activities and integration between
partners took place in less than two thirds of the NoEs assessed
(59 out of 101). Already in 2004, a previous report [20] acknowledged
the difficulties of NoEs in achieving the required ’durable integration‘. More recently, a study carried out on behalf of the Commission [21]
confirmed that only a minority of NoEs have moved convincingly
towards self-sustainable integration with prospects for longer-term
survival beyond the ending of EU funding.

**32.** NoEs often put in place only traditional forms of research collaboration
on individual actions, instead of coherent and long-term joint activities and partnerships.

**33.** Two main factors can explain the difficulty that NoEs have in achieving lasting integration. Firstly, the goal of setting up a new kind
of intra-European network, by integrating institutions previously in
competition with each other, requires a new approach to research
collaboration. The reluctance of many organisations to engage in a
long-term commitment did not favour this aim. Public research centres, at the heart of the NoEs’ objective, found difficulties in integrating with each other due to their institutional structure and budgetary
constraints. For industry, the treatment of intellectual property was
a matter of particular concern.

20 See Evaluation of the

effectiveness of the New

Instruments of Framework

Programme VI. Report

of the High-Level Expert

Panel chaired by Professor

R. Marimon, 21 June 2004

(http://cordis.europa.eu/fp6/

instruments_review/).

21 See Expert Group on the

future of Networks of Excellence,

Final Report, September 2008,

pp. 16 and 47.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

21

**34.** Secondly, there are areas where substantial integration can only be
achieved progressively. In practice, the maximal duration of five years
proved not to be realistic. However, in several cases (five of the eight
NoEs visited) the project duration had been even reduced during the
negotiation stage.

**35.** The fact that most NoEs financed under FP6 have not reached durable

integration raises the question as to the conditions under which it
is justified to continue their financing beyond the initial duration,
under the ongoing Seventh Framework Programme (FP7).

**LIGHT AND SHADE CONCERNING NoE PERFORMANCE**

One NoE assessed by the Court concerned a project in the field of information society technologies involving a consortium of a large number of both private (SMEs and large industries) and public participants
(universities). The consortium had been assembled in order to structure fragmented European research
in the specific scientific field, reduce duplication, boost excellence and spread scientific knowledge.

The Court reviewed the project’s achievements based on project documentation and interviews with
representatives of three participants audited. The opinion expressed by the independent experts who
carried out the project reviews was taken into account. The Court concluded that the project contributed to sharing new knowledge and to transferring pre-existing knowledge. Research collaboration
was promoted and new knowledge generated and disseminated through websites, the participation
and organisation of conferences, etc. This knowledge had potential commercial applications. A ‘Virtual
Centre of Excellence’(VCE) was created (a web portal built by project participants containing updated
information on public and private research in the specific scientific field). A school in the specific field
of the project has been created by several participants.

At the end of the project, the consortium requested additional funding for a second project phase,
which was obtained. The participants visited declared that their involvement in the integrating activities undertaken depended on the availability of additional EU funds. Once the funding ceased, it would
be impossible to maintain these activities.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

22

**IPS: LARGE PROJECTS, AND RELATIVELY LOW PARTICIPANTS’**
**FINANCIAL INVOLVEMENT**

**36.** Integrated Projects were designed to generate the knowledge required
to implement the thematic priorities, by achieving ambitious objectives of a European dimension.

**37.** The effectiveness of IPs was assessed by the Court for a sample of projects,
according to the following criteria:

—
Vertical integration of the full ‘value-chain’ of stakeholders, from
those involved in knowledge production through to technology
development and transfer.

—
Horizontal integration of a range of multidisciplinary activities.

—
Activity integration: integrating various research activities from
fundamental to applied research and with other types of activity, including take-up activities, protection and dissemination of
knowledge, and training.

—
Sectoral integration of actors from private and public sector
research organisations, and in particular between academia and
industry, including SMEs.

—
Financial integration of public and private funding.

23

**38.** The evidence shows that the audited IPs have achieved, in general, the
objectives pursued in their research field and have mobilised a significant volume of resources (human, financial, technical). This is
due in particular to a longer duration and an increased number of
participants when compared to pre-existing instruments [22] . However,
the audited IPs have not succeeded in attracting additional public
and private funding.

**SME PARTICIPATION TARGET NOT FULLY ACHIEVED**

**39.** SMEs play a key role in the European economy, given their weight in the
business sector and their importance in employment. As mentioned
in paragraph 8, FP6 sought to take their particular needs into account
and set for this purpose a target of at least 15 % of the total budget
for the FP6 thematic priorities, as compared to 10 % in FP5 [23] .

**40.** _**Figure 6**_ shows that, although more SMEs were involved in RTD projects
when compared to FP5, the EC contribution allocated to SMEs under
FP6 has decreased both in relative and absolute terms. The Commis
sion’s best estimate indicates that no more than 10 % of the total
budget for the FP6 thematic priorities was allocated to SMEs, representing some 1,2 billion euro. This is a lower rate than achieved
under FP5 Thematic Programmes (12 %, representing around 1,4 billion euro). FP6 has not succeeded in giving a significant stimulus to
SMEs’ RTD investment in the thematic priorities. Concerning FP7, first
results show that the participation of SMEs is further declining [24] .

**41.** There are objective difficulties in raising SMEs’ participation. They have
a more local/regional dimension and face by nature higher entry
barriers, mainly caused by the complex application procedures and
the costs of submitting proposals. Cooperation with large research
organisations can be hindered by higher financial and technical risks.
Also, like for private participants in general, SMEs are not always
prepared to subscribe to long-term contractual commitments for fear
of losing flexibility to react to changing needs.

22 Specific targeted projects.

23 In addition to the funding

foreseen under the thematic

priorities, FP6 included a specific

programme area intended to

support horizontal research

activities involving SMEs

(EUR 475 million).

24 Also FP7 has a similar target

for SMEs (at least 15 % of

the funding available under

the ‘Cooperation’ part of the

programme). Commission data

(October 2008) show that SME

participation accounts for 8,2 % of

project participations and 8,5 %

of the EC contribution awarded.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

24

**EVOLUTION OF FP PARTICIPATION (THEMATIC PRIORITIES)**

**BREAKDOWN OF FP6 PARTICIPANTS BY PROJECT INVOLVEMENT**

**(NUMBER OF INDIVIDUAL PARTICIPATIONS)**

|Col1|57 %|Col3|Col4|Col5|Col6|
|---|---|---|---|---|---|
|**53 %**<br>**57 %**<br>**37 %**<br>**28 %**<br>**8 %**<br>~~**13 %**~~<br>**2 %**<br>**1 %**|**57 %**|**57 %**|**57 %**|**57 %**|**57 %**|
|**53 %**<br>**57 %**<br>**37 %**<br>**28 %**<br>**8 %**<br>~~**13 %**~~<br>**2 %**<br>**1 %**|**3 %**|**3 %**|**3 %**|**3 %**||
|**53 %**<br>**57 %**<br>**37 %**<br>**28 %**<br>**8 %**<br>~~**13 %**~~<br>**2 %**<br>**1 %**|**37 %**|**37 %**|**37 %**|**37 %**||
|**53 %**<br>**57 %**<br>**37 %**<br>**28 %**<br>**8 %**<br>~~**13 %**~~<br>**2 %**<br>**1 %**|**28 %**|**28 %**|**28 %**|**28 %**||
|**53 %**<br>**57 %**<br>**37 %**<br>**28 %**<br>**8 %**<br>~~**13 %**~~<br>**2 %**<br>**1 %**||||||
|**53 %**<br>**57 %**<br>**37 %**<br>**28 %**<br>**8 %**<br>~~**13 %**~~<br>**2 %**<br>**1 %**|**8 %**<br>~~**13 %**~~|**8 %**<br>~~**13 %**~~|**8 %**<br>~~**13 %**~~|**8 %**<br>~~**13 %**~~||
|**53 %**<br>**57 %**<br>**37 %**<br>**28 %**<br>**8 %**<br>~~**13 %**~~<br>**2 %**<br>**1 %**|**2 %**<br>**1 %**|**2 %**<br>**1 %**|**2 %**<br>**1 %**|**2 %**<br>**1 %**||
|FP5<br>FP6<br>Public|FP5<br>FP6<br>Public|FP5<br>FP6<br>Private (non-SME)|FP5<br>FP6<br>Private (SME)|FP5<br>FP6<br>Undefned|FP5<br>FP6<br>Undefned|

**BREAKDOWN OF FP6 PARTICIPANTS BY EC CONTRIBUTION RECEIVED**

|Col1|58 %|Col3|Col4|Col5|Col6|
|---|---|---|---|---|---|
|**49 %**<br>**58 %**<br>**37 %**<br>**31 %**<br>**12 %**<br>**10 %**<br>**1 %**<br>**1 %**|**58 %**|**58 %**|**58 %**|**58 %**|**58 %**|
|**49 %**<br>**58 %**<br>**37 %**<br>**31 %**<br>**12 %**<br>**10 %**<br>**1 %**<br>**1 %**|**9 %**|**9 %**|**9 %**|**9 %**||
|**49 %**<br>**58 %**<br>**37 %**<br>**31 %**<br>**12 %**<br>**10 %**<br>**1 %**<br>**1 %**|**37 %**|**37 %**|**37 %**|**37 %**||
|**49 %**<br>**58 %**<br>**37 %**<br>**31 %**<br>**12 %**<br>**10 %**<br>**1 %**<br>**1 %**|**31 %**|**31 %**|**31 %**|**31 %**||
|**49 %**<br>**58 %**<br>**37 %**<br>**31 %**<br>**12 %**<br>**10 %**<br>**1 %**<br>**1 %**||||||
|**49 %**<br>**58 %**<br>**37 %**<br>**31 %**<br>**12 %**<br>**10 %**<br>**1 %**<br>**1 %**|**12 %**<br>**10 %**|**12 %**<br>**10 %**|**12 %**<br>**10 %**|**12 %**<br>**10 %**||
|**49 %**<br>**58 %**<br>**37 %**<br>**31 %**<br>**12 %**<br>**10 %**<br>**1 %**<br>**1 %**|**1 %**<br>**1 %**|**1 %**<br>**1 %**|**1 %**<br>**1 %**|**1 %**<br>**1 %**||
|F P5<br>FP6<br>Public|F P5<br>FP6<br>Public|FP5<br>FP6<br>Private (non-SME)|FP5<br>FP6<br>Private (SME)|FP5<br>FP6<br>Undefned|FP5<br>FP6<br>Undefned|

_Source_ : European Court of Auditors, based on Commission data.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

25

**42.** Moreover, some FP6 provisions did discourage the participation of private
companies. This was the case for the rule establishing consortia’s collective technical and financial responsibility towards the Commission
for the implementation of the project. This provision was applied to
private organisations but not to the public ones, creating _de facto_ an
additional burden for this target group. Collective financial responsibility has been removed under FP7 and replaced by a guarantee fund
financed by part of the advances paid to programme participants.

**43.** FP6 also included the automatic access to the participants’ pre-existing
knowledge required for the implementation of the project, unless
explicitly agreed otherwise. This provision resulted in a disincentive for private participation, in particular for SMEs which could not
compete on equal terms against larger organisations. Under FP7 more
freedom is given to consortia when defining access to existing knowledge in terms of Intellectual property rights.

**THE COMMISSION’S MANAGEMENT**
**OF NEW INSTRUMENTS**

**44.** The Commission’s management has a decisive role in ensuring the success
of FP6 activities, in particular by making it attractive to potential participants and steering project progress. Manageable structures, clarity
of objectives, rapid and straightforward procedures, and adequate
project monitoring are all determinants of the instruments’ effectiveness. In this context, the Court analysed the implementation of eight
calls for proposals and examined selected management process having a key impact on the instruments’ performance (namely negotiation
and project monitoring). The Court’s audit highlighted a number of
weaknesses.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

26

**UNCERTAINTIES ABOUT THE INSTRUMENTS SPECIFIC ROLE**

**45.** As shown by _**Figure 7**_, a number of actors (mainly participants and to
a lesser extent also coordinators) reached no or insufficient understanding about the specific objectives that the instruments were
to achieve and the way in which the instruments were supposed to
work. In particular for NoEs, the survey results indicate that they were
mainly understood as a networking instrument, while the objective
was that of favouring lasting integration among networked partners.
The start of FP6 was characterised by some confusion among potential applicants, particularly concerning NoEs. And further problems
were caused by inconsistent communication from the Commission
services [25] .

**46.** Understanding of policy objectives is a key factor for effectiveness. The
Commission has to ensure that different actors are clear about the
goals to be pursued. In this respect the role of the coordinators
should be highlighted, as they show a far better understanding than
participants. This indicates a problem of communication within consortia, probably aggravated by the high number of participants (see
paragraph 51).

25 See also the Expert Group on the

future of Networks of Excellence,

op.cit, Section 3.3 (p. 18) and

Annex 3 (p. 45).

27

**MULTIPLE LEGAL PROVISIONS**

**AND NOT ALWAYS SUFFICIENT GUIDANCE**

**47.** The legal provisions governing the instruments were spread across a
range of legal texts [26] . At different stages decisions were taken by
the Council and the European Parliament to set up the instruments
and their specific objectives (stating in which programme areas they
might be applied), or by the Commission concerning the provisions
for implementing the instruments (specifying in particular the basic
Community contribution mechanisms that were narrowly applicable
to each instrument) and finally the model FP6 contract (specifying
provisions for programme participants implementing indirect RTD
actions in the form of specific instruments). The different applicable
provisions were not always consistent.

**48.** Guidance was not complete when the first FP6 calls for proposals were
launched in December 2002. The delays affecting the preparation
and publication of guidelines were traceable, ultimately, to the late
adoption of model contracts.

**49.** In response to the publication of the 2004 evaluation [27], the Commission
took corrective action aimed at differentiating between the instruments. This resulted in a better understanding of the nature of the
instruments, but areas of misunderstanding persisted. As a result,
further guidance was provided on several aspects of FP6 project management not addressed by the initial programme guides [28] .

**THE NEGOTIATION PROCESS: MORE LENGTHY**

**AND OFTEN NOT TRANSPARENT**

**50.** Average time-to-contract [29] was about 13 months, to be compared with
a duration of nine months for FP5 calls [30] . As a result, consortia had
no choice but to start projects before the contract had been signed.
This entailed a legal and financial risk for the beneficiaries, which
is at odds with the objective of encouraging research activities. On
the other hand, the project scope (composition of the consortium,
duration, work programme) was modified in several cases (11 out of
14 projects audited) at the initiative of the Commission during the
negotiation process, without explicit reference to the objectives of
the programme.

26 See for example Annex III to

Decision No 1513/2002/EC, Annex

III to Council Decision 2002/834/EC

(OJ L 294, 29.10.2002, p. 1) and

Regulation (EC) No 2321/2002

of the European Parliament

and of the Council

(OJ L 355, 30.12.2002, p. 23).

27 See Evaluation of the

effectiveness of the New

Instruments of Framework

Programme VI, op. cit.

28 For example, the guide on

competitive calls for NoEs and IPs

appeared in November 2004. The

guide on Intellectual Property

rights appeared in March 2004.

29 Time to contract is defined

as the time elapsed between a

call closing date and a contract

signature.

30 See page 60 of Special Report

No 1/2004 on the management

of ‘indirect’ RTD actions under

the fifth framework programme

(OJ C 99, 23.4.2004).

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

28

**FEWER AND LARGER PROJECTS BUT STILL**

**A LARGE NUMBER OF PARTICIPANTS**

**51.** Although there is not a ‘once size fits all’ number of participants, it is
clear that the larger the number of participants, the more difficult the
project is to manage (organisational constraints, costs). Compared to
FP5, fewer contracts were signed under FP6 and they involved larger
consortia. However, on average, the number of participants per contract is higher in FP6 (NoEs involved on average 30 participants and
IPs around 25 partners). Under FP7 the number of participants has
been reduced (both NoEs and IP-like collaborative projects involve
on average slightly less than 20 partners).

**WEAKNESSES IN PROJECT MONITORING**

**52.** All indirect RTD actions implemented under FP6 were subject to periodic
technical monitoring by the Commission. For Networks of Excellence
and Integrated Projects, technical monitoring should be supported
by project reviews carried out by independent experts (see footnote
11). In particular, the Court found that two Commission departments
had failed to fully comply with the legal requirement to have annual
project reviews performed by independent experts.

**53.** Where the Commission’s departments did meet this requirement, harmonisation of project reviews was insufficient to allow the effective
use of review results as a performance indicator, at both project and
programme levels.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

29

**STIMULATION OF RTD INVESTMENT**

**54.** The total value of RTD investment carried out under FP6 thematic priorities amounted to 19 billion euro. The EU budget funded 63 % of this
investment. The private sector (including SMEs) contributed up to
22 %. The remaining 15 % was funded by national public sources [31] .

**MORE EU FUNDS FOR RELATIVELY**

**LESS PARTICIPANTS’ RTD INVESTMENT**

**55.** The comparison of FP6 (thematic priorities) with its predecessor (FP5
thematic programmes [32] ) shows that total investment and EU contribution have increased in absolute terms (by 8 % and 15 %, respectively).
However, the ratio ‘Total investment/EU contribution’ has declined
from 1,69 euro (FP5) to 1,59 euro (FP6). The main reason is that the
participation of the private sector in general has declined when compared to FP5 (see _**Figure 6**_ ).

**56.** The Court’s audit highlighted a risk of ‘low-profile’ commitment of public
bodies participating in the programme. Cases were found (12 out of
14 projects assessed) where the volume of own resources was much
lower (between 30 % and 80 % less) than the costs actually reimbursed
by the Commission [33] . In such cases, the stimulation of investment is
reduced, with the possibility that EU funds simply substitute participants’ own resources.

31 The only data available for

analysis correspond to the

contractually agreed estimated

eligible costs and the related

Community contribution. The lack

of more detailed information, in

particular the total investment

actually carried out as a result

of the participation in the

programme, makes impossible any

further analysis.

32 FP5 thematic programmes are

the predecessors of FP6 thematic

priorities. They were four: ‘Quality

of Life and Management of

Living Resources’, ‘User-friendly

Information Society’, ‘Competitive

and Sustainable Growth’ and

‘Energy, Environment and

Sustainable Development’.

33 According to the model applied

by a significant number of public

bodies (mainly universities),

participants should allocate

a volume of own resources

equivalent to those funded by

the Community. In this way the

principle of co-financing would be

satisfied. However, this condition

was not adequately monitored by

the Commission.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

30
#### CONCLUSIONS AND RECOMMENDATIONS

**57.** The Court recalls the lack in FP 6 of an explicit intervention logic, explaining how the different instruments and programmes were supposed
to strengthen the scientific and technological bases of Community
industry and to encourage it to become more competitive at international level [34] . The absence of SMART objectives and performance
indicators limits significantly the assessment of the contribution of
specific instruments and activities to the RTD objectives, thus undermining the basis for sound monitoring and evaluation (see paragraphs 19 and 20).

34 See paragraphs 22 to 37 of

Special Report No 9/2007.

**58.** The Court found that the FP6 instruments audited had promoted effective
research collaboration between project participants. However, while
all projects were qualified as ‘excellent’ at the time of their selection
by the Commission, not more than one third of them kept a rating
between ‘good to excellent’ after implementation (see paragraphs 22
to 27).

**59.** The assessment of NoEs’ and IPs’ specific objectives highlights a contrasting picture. Notwithstanding their success in promoting research
collaboration and projects of reasonable quality, none of the NoEs
audited achieved the specific objective of lasting integration among
networked partners, thus making future collaboration subject to continued public support. Among the difficulties there was the reluctance
of many organisations to engage in a long-term commitment, but
also the fact that the maximal duration of five years proved not to be
realistic, particularly in areas where lasting integration can only be
achieved progressively. This raises the question as to the conditions
under which it is justified to continue their financing beyond the
initial duration, under the ongoing Seventh Framework Programme
(FP7) (see paragraphs 29 to 35).

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

31

**60.** IPs have, in general, achieved the objectives pursued in their research
field. Thanks to a longer duration and an increased number of participants, they have mobilised a significant volume of resources.
However, IPs have not succeeded in attracting additional public and
private funding. (see paragraphs 36 to 38).

**61.** Overall, when compared to FP5, EU funds have increased in FP6 but the
investment realised is proportionally less important. The main reason
is that the participation of the private sector in general has declined
(see paragraph 55). The Court’s audit revealed cases where EU funds
were far higher than the own resources committed by public bodies
participating in the programme, with the possibility that EU funds
simply substitute participants’ own resources (see paragraph 56).

**62.** FP6 sought to take SMEs particular needs into account and set for this
purpose a target of at least 15 % of the total budget of the thematic
priorities (like in the current FP7). Although, when compared to FP5,
more SMEs were involved in RTD projects, the Community contribution allocated to SMEs in the thematic priorities was limited to 10 %
(against 12 % in FP5). FP7 first results show a further decline (see
paragraphs 39 to 40).

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

32

**63.** The Court observes that FP6 has not stimulated a significant increase
in terms of participants’ RTD investment, in a context where the
European Council has set the objective of reaching RTD investments
in the EU equivalent to 3 % of gross domestic product, two thirds of
which should come from the private sector.

**64.** The Commission’s management has a decisive role in ensuring success
and making schemes attractive to potential participants. Manageable
structures, clarity of objectives, rapid and straightforward procedures,
and project monitoring are all determinants of the Framework Programme’s effectiveness.

**65.** The Court’s audit highlighted a number of issues which affected the
effectiveness of the actions funded by FP6. These concern in particular
the uncertainty about the instruments’ specific role, the multiplicity
of legal provisions with guidance not always sufficient, a negotiation process that is increasingly long and often not transparent, the
rather large number of participants/grant and weaknesses in project
monitoring (see paragraphs 44 to 53).

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

33

This report was adopted by the Court of Auditors in Luxembourg at its
meeting of 4 June 2009.

_F_ _**o**_ _r_ _**t**_ _h_ _**e**_ _**Cou**_ _r_ _**t**_ _**o**_ _f A_ _**ud**_ _i_ _**to**_ _r_ _**s**_

Vítor Manuel da Silva Caldeira

_President_

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

**(iii)**
The Commission considers that its management of the FP6 instruments has played
an important role in ensuring the success
achieved by these instruments. The Commission has actively sought to address any
problems encountered, within the constraints imposed by the existing legislative
framework. Draft guidance documents were
actually available even before the launch of
FP6.

**(iv)**
FP6 alone (with its two new instruments, the
IPs and NoEs) is not the panacea for the challenges facing the European research landscape, especially since it made up only 5 %
of the overall European funding for research.
Moreover, at a moment when thousands of
projects financed by it continue their course,
it is too early to make conclusive comparisons with FP5. Further, the Commission
believes that FP6 has provided a significant boost to research efforts and that the

achievements of FP6 have been substantial.

The Commission acknowledges the Court’s
findings as regards the participation of the
private sector and would like to add that
when taking into account public and private
organisations engaged in commercial activities (the business enterprise sector) participation rates are similar to those of FP5.

The Commission agrees about the difficulties related to the SME participation under
FP6. These issues have been addressed in

FP7.

35

#### REPLY OF THE COMMISSION

**IV.**

The FPs have always been based on a sound
intervention logic which has allowed effective evaluation and monitoring. In FP7, the
intervention logic is made more explicit,
together with the objectives and performance measures that will help further
strengthen the evaluation and monitoring
system. In FP7, research goals and research
policy objectives are clearly and explicitly
set in the work programmes. For each topic,
the scientific goals, the scope of activities
and indications on the results expected are
defined.

**V.**

**(i)**

The Commission has evaluated IPs and NoEs

through various studies and reports. With
regard to a continued funding of FP6 NoEs
under FP7, the Commission reached a similar
conclusion to the Court.

**(ii)**
The Commission is constantly examining the
research landscape, monitors the national
R & D policies and is actively engaged in
endeavours to further research investments
through, not least, the open method of
coordination.

**(iii)**
The Commission agrees that clear and timely
guidance, a speedier contracting process and better project monitoring are very
important. The Commission carries out its
obligations while it constantly has to balance between two goals that are difficult
to reconcile, namely the speediness of disbursement of funds and the need to ensure

their accountable distribution to contractors, all in the context of the given legislative environment.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

**INTRODUCTION**

**10.**

FP6 was meant to encourage the participation of SMEs in all areas and all instruments,
in particular in the context of the activities
carried out in the priority thematic areas.

**16.**

A lower success rate for proposals under FP6
than under FP5 is attributed to the higher
subscription rates under FP6. Subscription
rates depend on a number of complex factors, among others the degree of detail of
the calls for proposals, the funds available
and the evaluation approach.

**AUDIT OBSERVATIONS**

**20.–21.**

In its response to a previous audit of the
Court [1], the Commission maintained that
the FPs have always been based on a sound
intervention logic, which has been articulated as per the Treaty’s provisions in the
various legislative and implementation acts.
In FP7 this is made more explicit, which will
further strengthen the Commission’s monitoring and evaluation. It is challenging, in
particular given the inherent uncertainty
of research results, to ensure all objectives
fit precisely within the strict definitions of
‘SMART’ criteria. Despite the fact that this
approach was not explicit in the Financial
Regulation applicable at the moment of
the FP6 preparation and adoption, essential elements of the ‘SMART’ approach
were nevertheless taken up in the Commission’s work programmes under FP6. Under
FP7, which was developed on the basis of
a detailed _ex ante_ impact assessment, all
work programmes contain expected impact
statements, which correspond with the
objectives, rationale and activities set out
in the Specific Programmes.

1 See the Commission’s replies to Special Report No 9/2007.

36

#### REPLY OF THE COMMISSION

**24.**

The Commission considers that the high
overall satisfaction with the two instruments

(NoEs and IPs) expressed by the participants
in the Court’s survey is very positive. Given
the different scope and profile of the two
instruments, it is normal that private partners found in particular IPs more suitable to
fit their needs.

**27.**

As proposals correspond to ‘expectations’
and projects to reality, and as the implementation of projects can result in unexpected difficulties, it is not surprising that
the evaluation of some projects could be
less positive than the evaluation of the proposals they stemmed from.

**30.**

The Commission notes that, according to Figure 5, the overall assessment of the projects’
quality showed them to be ‘acceptable’
and ‘good to excellent’ with a percentage
of above 80 %, which suggests that participants adequately allocated their research
capacities to the network. One of the most
important aspects of these new instruments,
repeatedly brought forward in various guidance documents, was the enhanced role of
the consortia themselves (and, in particular,
of the coordinators) concerning the internal
management of the project. It could be too
soon to reach a definitive conclusion on how
and/or if the objective of ‘self-sustainable
long-term research activities and partnerships’ has been reached.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

The Commission notes that the behaviour of

the NoEs has been variable across the Frame
work Programme. In some thematic priorities such as IST and NMP, NoEs tend to reach
greater degrees of sustainable development
than in others [2] .

**31.**

The Commission notes the Court’s own

statement that almost two thirds of NoEs
achieved full restructuring of activities and
integration between partners, which is a
considerable proportion.

**33.**

Certain circumstances may have hindered
participants from working together more
effectively. This is a normal trait, inextricably linked to the nature of collaboration
between various actors with complementary
yet different individual research goals.

**35.**
The possibility of continuing the financing of certain promising NoEs under FP7 is
addressed on a case-by-case basis. In fact, an
FP6 NoE’s duration may be prolonged under
FP7 to achieve its objectives but without new
funding (cases of these currently exist).

**38.**

A distinction has to be made between the
possible dimension of financial integration,
in the sense of the guidelines issued by the
Commission for IPs, and the co-financing
principle governing any FP6 project. There
has been no requirement on IP participants
to attract other sources of funding beyond
their own co-financing share. The Commission notes variable behaviour across the thematic areas of the Framework Programme in
terms of attracting additional private and
public funding.

2 IST and NMP refer to the FP6 thematic priorities ‘Informa

tion society technologies’ and ‘Nanotechnologies and nano

sciences, knowledge-based multifunctional materials, and

new production processes and devices’.

37

#### REPLY OF THE COMMISSION

**40.**

Concerning FP7, it is still too early in the course
of this seven-year long FP to suggest any final
judgment on the SMEs’ participation.

**41.–43.**
The Commission agrees about the difficulties related to the SME participation under
FP6. These issues have been addressed
in FP7. In addition, on top of the funding
provided following calls for proposals, FP7
supports the Eurostars Joint Programme for
SMEs based on Article 169 of the Treaty by
up to 100 million euro. The programme is
jointly undertaken by Eureka countries, supporting transnational projects initiated and
led by R & D performing SMEs. Eurostars is
expected to contribute to building the European Research Area by integrating the participating national programmes into a joint
programme at European level.

**44.**

The Commission considers that the Court’s

conclusion that the FP6 instruments audited

had promoted effective research collaboration between project participants (see paragraph 58) reflects well on the Commission’s
overall management of these instruments.
The Commission has actively sought to
address any problems encountered, within
the constraints imposed by the existing legislative framework, and continues to do so
under FP7.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

**45.**

The Commission provided guidance material
to the actors involved in order to facilitate
their understanding of the new instruments.
A very high percentage of coordinators and
participants in the survey apparently did
understand the distinction between the two

instruments (see the Court’s Figure 7). It is
also to be expected that coordinators would
have a better understanding of these than
other participants. The European research
community by and large understood and
used well the new instruments, as evidenced
by Figure 7 itself.

**46.**

One of the most important aspects of these
new instruments, repeatedly brought forward in various guidance documents, was
the enhanced role of the consortia them
selves (and, in particular, of the coordinators) concerning the internal management
of the project.

**47.**

The Commission implemented the FP in the
best possible manner to ensure compliance
with the different applicable provisions.

**48.**

The Commission has acted as quickly as possible given the late adoption of the FP6 legal
acts by the legislators. Detailed brochures
and ‘Provisions for implementing’ for all
the instruments were already available in
November 2002, well before the production
of the model contracts. A large number of
guidance documents have been produced
and published on the internet. Additionally,
targeted events for coordinators, to which
all projects have been invited, addressed in
particular any questions they might have
had on the specificities of the instruments.

38

#### REPLY OF THE COMMISSION

**49.**

The Commission is pleased that the Court
appreciates its continuous efforts to promote
better understanding of the new instruments
under FP6, including its quick response to
external evaluations. The European research
community by and large understood and
used the new instruments well.

**50.**

The introduction of new instruments unavoidably brought the need for proposers to
become familiar with new notions in terms
of the negotiations to be carried out and
the data to be submitted, elements that
very often made necessary further contacts
and exchanges between participants themselves, an always time-consuming process.
Further, the obligation imposed on the Commission by the legislator (the Council, in this
case) to receive a (positive) opinion on all
IPs and NoEs to be funded (irrespective of
the amount of Community contribution)
contributed to making the time-to-contract
longer. Moreover, the time is comparable
to that of other major funding agencies in
the world. As regards the negotiation process, modifications on the project scope were
always done in the context of the FP’s legal
framework.

**51.**

The Commission acknowledges the Court’s
comments. However, the Commission recalls
that one of the most important aspects of
these new instruments, repeatedly brought
forward in various guidance documents, was
the enhanced role of the consortia them
selves (and, in particular, of the coordinators) concerning the internal management
of the project.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

**52.**

The Commission strongly endorses the
importance of project monitoring, carried
out through a variety of means including
the detailed attention of project officers. It
recognises however, that there have been
some cases, often for individual and specific
reasons, where independent review was not
carried out.

**53.**

A common template for the reviewers’ report
was prepared for all Research DGs. A basic
common guidance document was also prepared and given to the different services to
adapt to their own needs.

**55.**

The Commission acknowledges the Court’s
findings. The Commission would like to
stress, however, that the EU contribution
to the private sector under FP6 increased
significantly by more than 1 billion euro in
comparison to FP5. It is also true that the
situation differs by sectors. The Commission
considers that FP6 had a positive influence
on industrial competitiveness. Furthermore,
it would like to add that when taking into
account public and private organisations
engaged in commercial activities (the business enterprise sector) participation rates
are similar to those of FP5.

39

#### REPLY OF THE COMMISSION

**CONCLUSIONS AND**
**RECOMMENDATIONS**

**57.**

The Commission considers that the FPs have

always been based on a sound intervention
logic. Under FP6, the legislative acts and the
Commission’s work programmes and information material of various types provided
a coherent stream and wealth of informa
tion on intervention logic, objectives, justification of the efforts and their European
added value, actions envisaged, rationale
and operational framework of application
for all instruments to be used and other

helpful information for the proposers and
project participants. It is challenging, in
particular given the inherent uncertainty
of research results, to ensure all objectives
fit precisely within the strict definitions of
‘SMART’ criteria. Despite the fact that this
approach was not explicit in the Financial
Regulation applicable at the moment of
the FP6 preparation and adoption, essential elements of the ‘SMART’ approach were
nevertheless taken up in the Commission’s
work programmes under FP6. Under FP7, all
work programmes contain expected impact
statements, which correspond with the
objectives, rationale and activities set out
in the Specific Programmes. The Commission
considers that FP6 was properly monitored
and evaluated.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

**Recommendation No 1**

The Commission considers that the FPs have

always been based on a sound intervention
logic which has allowed effective evaluation and monitoring. Under FP7, which was
developed on the basis of a detailed _ex ante_
impact assessment, all work programmes
contain expected impact statements, which
correspond with the objectives, rationale
and activities set out in the Specific Programmes. This will help further strengthen
the evaluation and monitoring system. Under
FP7, instead of being encapsulated in the
‘instruments’, research goals and research
policy objectives, including those aiming at
structuring research efforts, are clearly and
explicitly set in the work programmes. For
each topic on which proposals are invited,
the scientific goals, the scope of activities
and indications on the results expected are
defined.

**58.**

The Commission welcomes the Court’s con
clusion that the FP6 instruments covered

by the audit had promoted a good level
of research collaboration between project
participants. As proposals correspond to
‘expectations’ and projects to reality, and as
the implementation of projects can result in
unexpected difficulties, it is not surprising
that the evaluation of some projects could
be less positive than the evaluation of the
proposals they stemmed from.

40

#### REPLY OF THE COMMISSION

**59.**
There was a large number of NoEs where
lasting integration among partners was
achieved and the overall quality of projects
was deemed high (the overall assessment
in Figure 5 shows the NoEs’ quality to be
‘acceptable’ and ‘good to excellent’ with a
total percentage of above 80 %). Almost two
thirds of NoEs achieved full restructuring of
activities and integration between partners.
The possibility of continuing the financing
of certain promising FP6 NoEs under FP7 is
addressed on a case-by-case basis. In fact, an
FP6 NoE’s duration may be prolonged without new funding (cases of these currently
exist). Moreover, new NoEs could be envisaged in duly justified cases.

**60.**

A distinction has to be made between the
possible dimension of financial integration
and the co-financing principle governing any
FP6 project. There has been no requirement
on IP participants to attract other sources
of funding beyond their own co-financing share. The Commission notes variable
behaviour across the thematic areas of the
Framework Programme in terms of attracting
additional private and public funding.

**Recommendation No 2**

The Commission has evaluated these FP6

instruments through various means. With
regard to a continued funding of FP6 NoEs,
the Commission reached a similar conclusion
following the Report of the Expert Group on
the future of Networks of Excellence.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

**61.**

The Commission acknowledges the Court’s
findings. Still, it points out that this did not
diminish the FP’s role as a contributor to the

promotion of industrial competitiveness.

Furthermore, it would like to add that when
taking into account public and private
organisations engaged in commercial activities (the business enterprise sector) participation rates are similar to those of FP5.

**62.**

Concerning FP7, it is still too early in the
course of this seven-year-long FP to suggest any final judgment on the SMEs’
participation.

**63.**

FP6 makes up a very small part (close to 5 %)
of the overall funding provided for research
in Europe and, hence, its scope for influence in a domain where Member States have
the first role to play is by definition limited. Yet the Commission believes that FP6
has provided a significant boost to research
efforts.

**Recommendation No 3**

The Commission is constantly examining the
research landscape, monitors the national
R & D policies and is actively engaged in
the endeavours to further research invest
ments through, not least, the open method
of coordination.

41

#### REPLY OF THE COMMISSION

**65.**

The Commission has actively sought to
address any problems encountered, within
the constraints imposed by the existing legislative framework. The Commission notes
that the large number of projects under
the new instruments, which points to the
involvement of thousands of participants in
them, as well as the overall good results that
were registered for them, shows that any
shortcomings have not been important for
the overall use of the IPs and the NoEs.

**Recommendation No 4**

The Commission agrees that clear and timely
guidance, a speedier contracting process and better project monitoring are very
important elements for the implementation
of a European FP for research. The Commission carries out its obligations while it constantly has to balance between two goals
that are difficult to reconcile, namely the
speediness of disbursement of funds and the
need to ensure their accountable distribu
tion to contractors, all in the context of the
current legislative environment.

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

43

#### REPLY OF THE COMMISSION

European Court of Auditors

**Special Report No 8/2009**

**‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?**

Luxembourg: Publications Office of the European Union

2009 — 41 pp. — 21 � 29.7 cm

ISBN 978-92-9207-408-1

doi:10.2865/63596

Special Report No 8/2009 — ‘Networks of Excellence’ and ‘Integrated Projects’ in Community research policy: did they achieve their objectives?

**How to obtain EU publications**

**Publications for sale:**

- via EU Bookshop (http://bookshop.europa.eu);

- from your bookseller by quoting the title, publisher and/or ISBN number;

- by contacting one of our sales agents directly.
You can obtain their contact details on the Internet (http://bookshop.europa.eu)
or by sending a fax to +352 2929-42758.

**Free publications:**

- via EU Bookshop (http://bookshop.europa.eu);

- at the European Commission’s representations or delegations.
You can obtain their contact details on the Internet (http://ec.europa.eu)
or by sending a fax to +352 2929-42758.

THE SIXTH FRAMEWORK PROGRAMME WAS SET UP BY THE EU TO

PROMOTE EUROPEAN RESEARCH AND TECHNOLOGICAL DEVELOPMENT

DURING THE PERIOD FROM 2002 TO 2006. THE COURT ANALYSED

THE EXTENT TO WHICH THE TWO MAIN INSTRUMENTS, ‘NETWORKS

OF EXCELLENCE’ AND ‘INTEGRATED PROJECTS’, HAD EFFECTIVELY

CONTRIBUTED TO THE ACHIEVEMENT OF THE OBJECTIVES OF THE

PROGRAMME. THE COURT CONCLUDED THAT THESE INSTRUMENTS

CONTRIBUTED TO THE GENERAL OBJECTIVE OF PROMOTING EFFECTIVE

RESEARCH COLLABORATION BUT THEIR SPECIFIC OBJECTIVES WERE

ONLY PARTIALLY ACHIEVED.

EUROPEAN COURT OF AUDITORS