REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS First interim evaluation of the ARTEMIS and ENIAC Joint Technology Initiatives
Date: 2010-12-16
Institutions: European Commission
Subjects: EU programme, EU research policy, joint venture, public-private partnership

EN 

EN 

   EN 

 
EUROPEAN COMMISSION 

Brussels, 16.12.2010 
COM(2010) 752 final 

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE 
COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE 
COMMITTEE OF THE REGIONS 

First interim evaluation of the ARTEMIS and ENIAC Joint Technology Initiatives 

EN 

   EN 

 
 
  
REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE 
COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE 
COMMITTEE OF THE REGIONS 

First interim evaluation of the ARTEMIS and ENIAC Joint Technology Initiatives 

1. 

INTRODUCTION 

This  report  highlights  the  findings  and  recommendations  of  the  first  interim  evaluation  of 
ARTEMIS  and  ENIAC.  They  are  the  two  Joint  Undertakings  implementing  the  Joint 
Technology Initiatives (JTIs) in the fields of embedded systems and nanoelectronics research. 
The report formulates the Commission’s initial response to the evaluation’s recommendations 
and sets out follow-up measures. Thereby the Commission complies with the requirement to 
report on the first interim evaluation of ARTEMIS and ENIAC as laid down in their founding 
acts1.  Furthermore,  the  Commission  intends  to  prepare  a  Communication,  scheduled  for  the 
first half of 2011, to set out its overall conclusions on the interim evaluations of all the JTIs 
and the interim assessments of the Recovery Plan Public-Private Partnerships (PPPs). 

2. 

BACKGROUND 

In  its  current  Seventh  Framework  Programme  for  research,  technological  development  and 
demonstration  activities,  the  European  Union  supports  a  number  of  Joint  Technology 
Initiatives  (JTIs).  In  the  field  of  Information  and  Communication  Technologies  (ICT),  the 
ARTEMIS and ENIAC JTIs were established in 2007 as Joint Undertakings (JUs), based on 
Article  187  TFEU  (ex  Article  171  EC).  These  two  bodies  constitute  public-private 
partnerships between industry, a number of EU Member States and associated countries, and 
the European Union. Their aim is to implement, by means of a budget from both the EU and 
participating  Member  States2,  a  research  agenda  defined  by  the  European  research 
communities  (industry  and  academic/research  organisations)  in  their  respective  fields.  The 
JTIs  thereby  seek  to  strengthen  Europe’s  future  growth,  competitiveness  and  sustainable 
development. Their ambition and scope, the scale of the financial and technical resources that 
need to be mobilised, and the need to achieve effective coordination and synergy of resources 
and funding called for action at European level. 

After  their  first  two  calls  for  proposals,  ARTEMIS  and  ENIAC  have  launched  25  and  18 
projects, respectively, each with an average duration of three years. The EU and JTI Member 
 576 m to both JTIs combined (2008-2010), in addition 
States together have so far committed €

1 

2 

Council  Regulation  No  74/2008  of  20  December  2007  on  the  establishment  of  the  ‘ARTEMIS  Joint 
Undertaking’ to implement a Joint Technology Initiative in Embedded Computing Systems; Article 11 
Council Regulation No 72/2008 of 20 December 2007 setting up the ENIAC Joint Undertaking, Article 
11. 
ARTEMIS  Member  States:  Austria,  Belgium,  Cyprus,  Czech  Republic,  Germany,  Denmark,  Estonia, 
Spain,  Finland,  France,  Greece,  Hungary,  Ireland,  Italy,  Latvia,  Netherlands,  Norway,  Portugal, 
Romania, Sweden, Slovenia, United Kingdom 
ENIAC Member States: Austria, Belgium, Czech Republic, Germany, Estonia, Spain, Finland, France, 
Greece,  Hungary,  Ireland,  Italy,  Latvia,  Netherlands,  Norway,  Poland,  Portugal,  Romania,  Slovak 
Republic, Sweden, United Kingdom. 

EN 

2 

   EN 

                                                 
 
to private R&D efforts worth about double that amount. While this represents a considerable 
public investment in two crucial technology fields, the present public support falls far short of 
the  original  objectives  set  in  the  Commission’s  proposals  for  establishing  the  Joint 
Undertakings (total of €
 900 m combined for the same period). The current investments in the 
two  JTIs  also  do  not  live  up  to  the  expectations  of  the  research  stakeholders,  who  had 
ambitions  for  programmes  amounting  to  €
 3  billion  each  (corresponding  to  the 
Commission’s overall proposal). 

 2.5  to  €

3. 

CONDUCT OF THE EVALUATION 

The objective of this evaluation was to assess the quality and efficiency of the JTI operations, 
as well as the likelihood that ARTEMIS and ENIAC, in their current set up, would achieve 
their objectives. To this end, the Commission invited a panel of independent experts3, chaired 
by Dr Wulf Bernotat, to review evidence and interview stakeholders4. 

The  evaluation  panel  issued  its  report  in  July  2010.  The  full  report  is  available  on  the 
EUROPA website5. 

4. 

THE EVALUATION RESULTS AND RECOMMENDATIONS 

Significantly,  the  evaluation  panel  recognises  the  value  of  a  tripartite  structure  for  JTIs  in 
these fields, pooling resources from industry, the EU and Member States: 

‘Having a joint strategy with shared implementation is good for industry, good 
for  Member  States,  and  good  for  Europe.  (…)  The  Evaluation  Panel 
congratulates  all  those  involved  for  their  considerable  achievements  in  the 
design and implementation of these new instruments.’ 

However,  the  experts  express  their  concern  that  ARTEMIS  and  ENIAC  are  not  giving 
sufficient attention to their European strategic aims. The panel calls for a strategic re-focusing 
of the two JTIs, involving all stakeholders and moving further towards a truly joint effort. In 
this context, the experts further demand that industry re-engage the top-level ‘thought leaders’ 
in industry, governments and the scientific community: 

‘Industry,  that  asked  for  and  was  given  leadership  of  these  JTIs,  has  not 
maintained its initial engagement in the strategic direction and management of 
the  JTIs  so  that  they  achieve  all  their  objectives  and  specifically  has  not 
engaged effectively with Member States to address the problems.’ 

3 

4 

5 

The  expert  panel  consisted  of  the  following  members:  Dr  Wulf  H.  Bernotat  (chairman)  —  formerly 
E.ON  AG;  Elke  Eckstein  —  OSRAM  Opto  Semiconductors;  Luke  Georghiou  —  University  of 
Manchester, Manchester Institute for Innovation Research; Terttu Luukkonen — the Research Institute 
of the Finnish Economy; Bob Malcolm (rapporteur) — Ideo ltd; Dominique Potier, Pôle Systématique 
and  formerly  Thales;  Christian  de  Prost,  ATMEL;  Alberto  Sangiovanni-Vincentelli,  University  of 
California/Berkeley. 
The evidence base for this evaluation included the legal and other documents setting up ARTEMIS and 
ENIAC,  financial  information,  participation  statistics  and  project  information.  Interviewees  included 
representatives from the JUs, the Commission, Member State authorities and the business and research 
communities, including SMEs. 
http://ec.europa.eu/dgs/information_society/evaluation/rtd/jti/index_en.htm. 

EN 

3 

   EN 

                                                 
The evaluation panel considers that the funding shortfall is a critical issue: 

‘The funding commitment by Member States is significantly below that which 
was expected, jeopardising the ability to establish a critical mass of activity and 
severely constraining the construction of appropriate portfolios of projects.’ 

The  shortfall  appears  to  be  partly  related  to  the  ITEA2  and  CATRENE  Eureka  inter-
governmental  schemes,  which  work  in  parallel  with  ARTEMIS  and  ENIAC  and  which  are 
supposed  to  progressively  integrate  with  the  JTIs  where  added  value  can  be  created6.  The 
Commission’s position on this problem is outlined in section 6 of this report. 

Addressing the European Commission, the evaluation panel highlights a number  of lessons, 
such as the need for greater organisational and financial flexibility, which have been learned 
with the first generation of JTIs and should be taken into account when setting up potential 
future PPPs: 

‘The present financial regulations and other administrative requirements (such 
as  staff  regulations)  are  too  constraining  and  they  inhibit  rather  than  enable 
realisation of the strategic aims for JTIs.’ 

The list of the panel’s recommendations is reproduced in annex. 

The  timeframe  for  each  recommendation  indicates  whether  it  is  possible  to  act  under  the 
current legal framework or if the existing JU Regulations or Financial Regulation need to be 
amended.  As  the  amendment  of  the  legal  framework  is  a  resource-intensive  procedure,  this 
group of recommendations will be dealt with as part of the discussion about a next generation 
of PPPs. 

5. 

PLANNED FOLLOW-UP ACTIONS 

The  Commission  welcomes  the  interim  evaluation  report  and  takes  note  of  the  evaluation 
panel’s  recommendations.  The  Commission  is  committed  to  address  its  part  of  the 
recommendations  and  to  work  with  industry,  Member  States  and  the  Joint  Undertakings  to 
help  them  address  their  parts.  In  line  with  the  timeframe  suggested  by  the  report,  and 
considering the specific nature of ARTEMIS and ENIAC directly involving Member States, 
the Commission will undertake:  

1) immediate actions for the existing Joint Undertakings (i.e. the JTI activities in the period 
2011-2013) to be followed up through their governance structures; and  

2)  longer-term  actions  for  the  potential  next generation  of  PPPs  in  these  domains  under  the 
Innovation Union7 and Digital Agenda for Europe8 flagship initiatives launched to address the 
Europe 2020 strategy9. 

The Commission’s response to the panel recommendations is summarised below. 

6 
7 
8 
9 

Article 2(d) of Council Regulations No 72/2008 and No 74/2008. 
COM(2010) 546, http://ec.europa.eu/research/innovation-union 
COM(2010) 245, http://ec.europa.eu/information_society/digital-agenda 
COM(2010) 2020,. http://ec.europa.eu/europe2020 

EN 

4 

   EN 

                                                 
(i) General recommendation 

Notwithstanding the currently modest collective financial commitment by the Member States 
to  the  existing  JUs,  which  affects  directly  the  European  funding10,  and  the  apparent 
complexity, the Commission agrees that the tripartite model can bring important benefits if all 
parties  are  working  towards  a  strong  common  set  of  strategic  objectives.  Besides  industrial 
leadership  in  the  development  of  the  strategy,  this  model  offers  a  unique  opportunity  to 
coordinate  national  policies  towards  common  European  challenges.  At  the  same  time,  the 
Commission  agrees  that  the  JTIs  would  benefit  from  recommitment  by  the  partners  to  the 
tripartite model and to a European strategic agenda as soon as possible. The Commission will 
engage  with  industry  and  Member  States  to  explore  the  best  way  to  achieve  this.  This  is 
particularly  important  because  the  JTIs  are  active  in  areas  identified  as  Key  Enabling 
Technologies for Europe11. 

Without  prejudice  to  future  decisions  on  the  architecture  of  Framework  Programme  8,  the 
Commission  does  not  rule  out  that  activities  in  areas  addressed  by  ARTEMIS  and  ENIAC 
might be dealt with by a single JTI in the future. This might even cover a larger part of the 
value chain, addressing the burning issue of the trade-off between hardware and software and 
balancing  technology-push  and  application-pull  priorities.  In  addition,  this  would  help 
increase the efficiency of the operational set-up. 

(ii) Recommendations for Member States 

Given the strict link binding the European contribution to national contributions (see footnote 
10), the Commission acknowledges that there is a need to increase the financial contributions 
from  the  Member  States  in  order  to  reach  the  overall  financial  objectives  set  out  in  the 
Regulations12. In addition, the national budgets could cover a multi-annual period in order to 
set  a  longer-term  perspective  following  industry’s  multi-annual  strategic  plan.  The 
Commission  will  propose  that  the  Member  States  agree  on  a  multi-annual  budgetary 
contribution for the remaining lifetime of the JUs with the aim of achieving the original goals 
of the JTIs. This is one element of the strategic recommitment the Commission expects from 
its partners. 

Member  States  have  so  far  paid  little  attention  to  the  required  synchronisation  of  their 
operations13.  The  Commission  will  insist  that  national  practices  be  adopted  in  line  with  the 
terms  of  the  Regulations  and  the  Administrative  Arrangements  in  place  in  order  to  help 
converge on common practices. The Commission will also propose the creation of a Working 
Group  within  the  Public  Authorities  Boards  to  deal  with  the  benchmarking  of  national 
practices to improve the harmonisation of administrative processes and national funding rates. 

10 

11 

12 

13 

Article  13(4)  of  the  statutes  of  the  Joint  Undertakings  (annexed  to  Council  Regulations  72/2008  and 
74/2008)  states  that  a  JU’s  contribution  (using  the  EU  contribution)  to  the  budget  for  each  call  for 
proposals must be equivalent to 55 % of the total amount committed by the Member States. 
‘Preparing  for  our  future:  Developing  a  common  strategy  for  key  enabling  technologies  in  the  EU’ 
COM(2009) 512. 
Article  11(6)  of  the  statutes  of  the  Joint  Undertakings  (annexed  to  Council  Regulations  72/2008  and 
74/2008),  amounting  to  €
 738 m  for 
ARTEMIS. 
Article 13(6)(b) of the statutes of the Joint Undertakings (annexed to Council Regulations 72/2008 and 
74/2008): ‘… Member States shall undertake best efforts to synchronise the terms and conditions and 
the establishment of grant agreements and to disburse their financial contributions in a timely manner’.. 

 792 m  public  funding  for  ENIAC  and  €

 410 m  +  €

 440 m  +  €

EN 

5 

   EN 

                                                 
In this context, the funding rules for universities and academic/research organisations deserve 
special attention. 

The  Commission  agrees  that  there  is  value  in  making  more  visible  the  national  support  for 
specific  topics  in  the  Annual  Work  Programme.  However,  the  European  nature  of  the  JTIs 
should  not  take  the  form  of  a  collection  of  purely  national  interests.  The  Commission  will 
propose in the Public Authorities Board to develop a simple way to include national support 
considerations at an appropriate level of detail in the Annual Work Programme. 

(iii) Recommendations for the industrial associations 

The Commission agrees that adequate processes and tools should be put in place in order to 
monitor  and  analyse  the  strategic  achievements  of  the  JTIs.  It  will  propose  measurable 
indicators and ask the industrial associations to regularly undertake a strategic analysis of the 
results and impact of each call for proposals. More generally, the Commission will suggest to 
industry that strategic research agendas be updated and revised in due time and adopted at a 
high level in the companies involved, the Commission and Member States. 

The Commission supports the view that the industrial associations have a key role to play in 
driving the JTI strategy towards innovation ecosystems. The industrial partners  should steer 
the  development  of  a  concrete  plan  with  tangible  targets  which  the  JUs  would  have  to 
implement, such as stimulating the participation of SMEs, clustering activities, and education 
and training actions. 

Furthermore, the industrial associations are encouraged to provide more ‘value for money’ to 
their  members,  which  should  result  in  a  broader  representative  set  of  stakeholders  and  a 
broader membership with a particular focus on SMEs. The Commission will also analyse the 
potential and legal implications of a simpler administrative involvement of some participants 
in  the  projects,  for  example  the  concept  of  ‘associated  partner’  for  SMEs  with  a  view  to 
reducing their administrative and financial burden. 

(iv) Recommendations for the European Commission 

The  evaluation  panel  calls  on  the  Commission  not  to  apply  the  framework  Financial 
Regulation or the Staff Regulations to potential future PPPs. This is also broadly in line with 
the  recommendations  made  by  the  JTI  Sherpas'  Group14.  In  this  context,  the  Commission 
proposed,  in  the  triennial  revision  of  the  Financial  Regulation15,  two  additional  options  to 
implement public-private partnerships: a mixed public-private body based on Article 185a of 
the  Financial  Regulation  and  a  private-law  body  (Article  53(1)(2)(g).  This  proposal,  if 
adopted by the legislator, would create the framework for following this recommendation. 

The Commission will explore the relevance of extending the operational activities of potential 
to  encompass  accompanying  measures  such  as  coordination,  support, 
future  PPPs 
infrastructure  and  large  demonstration  actions.  It  will  also  investigate,  with  the  help  of  the 
Member  States,  industry  and  other  potential  contributors,  ways  of  combining  funding  from 
different sources. Similarly, the Commission will consider how to provide financial support to 
activities  that  are  critical  to  achieve  the  European  strategic  objectives  of  the  JTIs  and  that 

14 

15 

Designing together the 'ideal house' for public-private partnerships in European research – JTI Sherpas' 
Group – Final Report - January 2010, http://cordis.europa.eu/fp7/jtis/ 
COM(2010) 260, http://ec.europa.eu/budget/documents/financial_regulation_en.htm 

EN 

6 

   EN 

                                                 
could  not  be  supported  under  the  current  JTI  set-up,  such  as  infrastructure  or  innovation 
projects that are not R&D activities in the strict sense. 

The Commission sees the contribution of the industrial associations to the running costs of the 
JUs as an important incentive for them to ensure good governance of the JU. It also takes note 
of  the  difficulty  experienced  by  the  associations  to  collect  adequate  membership  fees.  A 
future  model  in  which  each  beneficiary  would  pay  a  ‘fee’  to  the  JU  would  need  to  be 
examined with great care. 

The Commission also recognises the importance of gathering data to demonstrate the impact 
of  the  JTIs’  work,  which  would  be  used  in  future  evaluations.  To  this  end,  it  will  establish 
with the help of experts a set of indicators to assess the achievements of the JTIs against the 
goals enshrined in their founding acts16 and their multi-annual strategic plans. The resulting 
data will be analysed yearly. 

(v) Recommendations for the Joint Undertakings 

The  Commission  is  aware  of  the  difficulties  experienced  by  the  industrial  associations  in 
generating sufficient income from their membership fees or other types of activities without 
mandatory  contributions  from  all  beneficiaries  of  the  projects.  On  the  other  hand,  the 
Commission  considers  that  the  industrial  associations  have  the  potential  to  generate  extra 
revenue through a broader service offering to their members. The low income of the industrial 
associations  is  also  due  to  the  smaller  volume  of  the  programmes  because  of  lower 
contributions from the Member States than originally expected. Besides its efforts to revitalise 
the commitment of stakeholders and to encourage the industrial associations to diversify and 
extend  their  activities  and  membership,  the  Commission  will  support  steps  by  the  JUs  to 
improve the current situation. 

The  Commission  concurs  with  the  need  to  improve  the  match  between  the  portfolio  of 
supported  projects  and  the  strategic  European  aims  of  the  programme.  Therefore,  it  will 
propose to establish a working group to review the current evaluation criteria in order to better 
assess the way in which the projects contribute to the overall European aims of the JTIs and to 
enriching the innovation ecosystem. 

Furthermore,  the  Commission  will  support  the  establishment  of  open  and  transparent 
processes by relevant stakeholders to provide early, more visible and constructive feedback to 
applicants on the prospects for support from the Member States. 

6. 

THE  COMMISSION’S  POSITION  TOWARDS  EUREKA  AND  THE  PROBLEM  OF  UNDER-
INVESTMENT 

The  Commission  agrees  that  the  parallel  operation  of  Eureka  clusters  (CATRENE  and 
ITEA217) and the JTIs in the areas of nanoelectronics and embedded systems adds complexity 
to the European Research Area, is confusing for the research community and is intrinsically 
inefficient.  Moreover,  the  Council  had  called  for  ‘progressive  integration’  of  these  two 
funding mechanisms. 

16 
17 

Article 2 of Council Regulations 72/2008 and 74/2008. 
See: http://www.catrene.org and http://www.itea2.org. 

EN 

7 

   EN 

                                                 
The  Commission  fully  supports  retaining  a  top-down,  strategic,  pan-European  and 
technological  role  for  the  JTIs,  with  funding  to  implement  their  programmes,  but  also 
recognises  the  added  value  of  a  complementary,  bottom-up,  shorter-term  business-  and 
market-oriented  programme  involving  the  collaboration  of  a  few  companies  and  funded  to 
implement specific projects. However, the Commission considers that both approaches could 
be  implemented  under  one  single  public-private  partnership,  which  would  also  allow  for 
significant economies in the running of operations. To date, all stakeholders have made great 
efforts to identify ways of delineating the two types of programmes. The resulting situation is 
sub-optimal. Experience has shown that it is very difficult to exclude topics from one or other 
programme  when  exploring  detailed  options  beyond  high-level  principles.  Indeed,  a  large 
number  of  research  subjects  are  addressed  by  both  programmes  (in  the  application  and 
technology  fields).  This  should  not  come  as  a  surprise  when  considering  that  most  private 
founding  partners  are  common  to  both  sides.  Continued  co-existence  would  not  ensure  the 
coordination  and  synchronisation  of  the  two  independent  initiatives  or  offer  budgetary 
flexibility  for  Member  States  when  allocating  their  resources  to  the  different  schemes  (for 
those who are actively involved in both, which is the case for the largest contributors). 

The  Commission  therefore  considers  that  the  co-existence  of  Eureka  clusters  and  the  JTIs 
contributes to the present shortfall in contributions from the Member States to the JUs and the 
Eureka  clusters,  as  the  financing  of  both  instruments  often  comes  from  a  single  source  at 
national level. This important issue is not addressed in the report. The Council, however, had 
acknowledged this situation when adopting the regulations establishing the JTIs, in particular 
in  calling  for  ‘progressive  integration’  of  the  two  funding  mechanisms.  Integrating  the  two 
approaches under one roof would facilitate the implementation of a single strategy for Europe, 
ensure sufficient resources to reach critical mass and enable the partners involved to choose 
the  right  mix  of  instruments  for  their  aims,  i.e.  through  flexible  use  of  the  top-down  and 
bottom-up mechanisms. 

Even though the panel states that operational integration should only be considered over the 
long term and not for the period from 2014 onwards, the Commission’s ambition is to keep 
working  on  the  progressive  integration  of  these  two  funding  mechanisms.  The  Commission 
will  actively  participate  in  the  existing  structures  (JTI  and  Eureka)  and  working  groups  in 
order to make further progress with the differentiation and coordination of activities, and will 
study  the  detailed  conditions  under  which  the  progressive  integration  of  the  Eureka  clusters 
within the JUs’ operations would create added value. 

EN 

8 

   EN 

Annex - List of recommendations by the evaluation panel 

No 

Summary of recommendations 

Timeframe 

1 

Future JTIs in these domains to continue the tripartite JTI model 

Next-gen JTIs 

Recommendations for Member States 

2  Make multi-annual budgetary commitments 

3 

Comply with the JTI Council Regulations 

4  Undertake benchmarking & alignment studies of national practices 

5  Give early annual indication of support for specific topics 

Recommendations for industrial associations 

6 

7 

Lead the establishment of processes to monitor progress toward JTI 
objectives 

Lead the preparation of action plans for achievement of innovation 
ecosystem aims 

8 

Engage better with the JTI constituencies 

Recommendations for the European Commission 

Now 

Now 

Now 

Now 

Now 

Now 

Now 

9 

Lead the drafting of new Council Regulations with alternative 
Financial Regulations and Staff Regulations 

Next-gen JTIs 

10  Regulations should allow JTIs to support innovation-related activities 

Next-gen JTIs 

other than R&D 

11  Regulations should allow JTIs to accept funding from other sources 

Next-gen JTIs 

12  Regulations should allow the EU to make additional financial 

Next-gen JTIs 

contributions for strategic purposes 

13  Regulations should allow the Joint Undertakings to claim some of their 

Next-gen JTIs 

operational costs from non-members 

14  The Commission should establish data gathering to support assessment 

Now 

of the benefits of these JTIs 

Recommendations for the Joint Undertakings 

15  Establish a mechanism for recovering some of their operational costs 

Next-gen JTIs 

from non-member beneficiaries of the JTIs 

16  Place greater emphasis on strategic, European aims in proposal 

Now 

evaluation & selection processes  

17  Establish processes to give early feedback to proposers 

Recommendation for JTI–EUREKA coordination 

18  ARTEMIS & ENIAC should continue their initiatives to differentiate 
from and coordinate with ITEA2 and CATRENE, respectively 

Now 

Now 

EN 

9 

   EN 

 
 
 
EN 

10 

   EN