REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS Second interim evaluation of the ARTEMIS and ENIAC Joint Technology Initiatives
Date: 2013-11-27
Institutions: European Commission
Subjects: EU programme, European Joint Undertaking, computer system, innovation, nanotechnology, research and development, technological change













EUR-Lex - 52013DC0830 - EN











|



52013DC0830

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS Second interim evaluation of the ARTEMIS and ENIAC Joint Technology Initiatives /* COM/2013/0830 final */









REPORT FROM THE COMMISSION TO THE
EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE
AND THE COMMITTEE OF THE REGIONS
Second interim evaluation of the ARTEMIS
and ENIAC Joint Technology Initiatives
(Text with EEA relevance)
1.           Introduction
This
report highlights the findings and recommendations of the panel of independent
experts who conducted the second interim evaluation of the ARTEMIS and ENIAC
Joint Technology Initiatives (JTIs). It furthermore provides the Commission's observations
and sets out follow-up measures. ARTEMIS and ENIAC are the Joint Undertakings (JUs)
implementing JTIs in the respective fields of embedded computing systems and
nanoelectronics. In this manner, the Commission complies with the evaluation
requirements as laid down in Article 11.2 of the founding acts of the JUs[1]. 
2.            Background
The
JTIs, set up in the form of JUs under Article 187 of the Treaty on the
Functioning of the EU (former Article 171 of the Treaty), were introduced in
the Seventh Framework Programme (FP7) to support key areas of research and
technological development of importance to Europe’s competitiveness. The
ARTEMIS and ENIAC JUs were launched in February 2008.
The
ARTEMIS and ENIAC JUs are public-private partnerships between industry, a
number of EU Member States and associated countries (JTI member States)[2], and the European
Union with the specific aim to define and implement a common Research Agenda by the European
research communities (industry and academic/research organisations), achieve synergy and coordination of European
R&D, promote the involvement of SMEs and create
significant economic and social benefits. The main instrument is the
mobilisation of funds from the EU, JTI member States and industry.
Since their establishment, the ARTEMIS and
ENIAC JUs launched and evaluated, respectively, six (one per year) and nine
calls for proposals (one in each of 2008, 2009 & 2010 and two in each of
2011, 2012 and 2013). By the end of 2012, 102 projects had been funded by the ARTEMIS and ENIAC JUs. The
EU and JTI member States together have so far committed over €1.115 billion to both JTIs combined
(2008-2012), in addition to private R&D efforts worth more than €1.670
billion.
The 52 ARTEMIS projects
to date represent a total R&D investment of €990 million, comprising of €288
million national contributions, €163 million EU contribution and €539 million
industry contribution. Those projects address all eight Sub-Programmes of the
ARTEMIS Strategic Research Agenda in the areas of safety, healthcare, smart
environments, manufacturing, computing, security, sustainability and embedded
systems design. After a relative low commitment by ARTEMIS member States in the
first four years, the
introduction of the ARTEMIS Innovation Pilot Projects in 2012 gave a boost to
the JU activities resulting from higher commitments by member States.
In
the ENIAC case, the 50 projects to date represent a total R&D investment of
€1.795 billion, comprising of €382 million
national contributions, €283 million EU contribution and €1.130 million
industry contribution. The projects cover all eight work areas of the ENIAC Strategic Research
Agenda in the fields of automotive/transport, communications/lifestyle, energy,
health, safety/security, semiconductor design, semiconductor manufacturing and
the underlying equipment/materials. Since 2011, the ENIAC JU became an
important vehicle for the implementation of manufacturing pilot lines following
the recommendations of the High-Level Group on Key Enabling Technologies. This created
a steep increase in the joint investments by the stakeholders, bringing the
execution of the programme close to budget that was foreseen.
As
foreseen in the Council Decisions establishing
the ARTEMIS and ENIAC JU, the Commission has to carry out, by 31 December 2010
and by 31 December 2013, an interim evaluation with the assistance of
independent experts. The evaluations cover the ARTEMIS and ENIAC JU quality, efficiency
and progress towards the objectives set. The first interim evaluation of the
ARTEMIS and ENIAC JTIs[3] was carried out after two years of the initiatives launch and it
mainly assessed the efficiency, management and operations of the JUs. The main objective
of the second interim evaluation is to assess the achievements of the ARTEMIS
and ENIAC JUs after more than 5 years of operation.
3.            Evaluation process
The
second interim evaluation examined the relevance, effectiveness, efficiency and
research quality of the JUs. In addition, it performed a follow-up of the implementation
of the recommendations of the first interim evaluation in 2010 and to the
Commission's report[4] on this interim
evaluation report. 
The
Commission appointed a panel of external and independent experts, chaired by
Dr. Götzeler[5]. In the composition of
the panel care was taken to ensure a good coverage of the technical domains as
well as continuity with the panel in charge of the first interim evaluation. The
panel operated by consensus and based its findings and recommendations on extensive
desk research, peer review evidence and interviews with stakeholders[6].
The
evaluation by the experts was conducted between September 2012 and February
2013. Their final report was issued in July 2013 and is available on the
Digital Agenda website[7]. The evaluation ran in
parallel with the drafting of the Commission proposal on the setting up of a
new JU to replace and succeed the ARTEMIS and ENIAC JUs. The conclusions and
recommendations of the evaluation of ARTEMIS and ENIAC JUs were inter alia analysed
and served as input to the Impact Assessment for the new JTI’s regulation.
4.            The evaluation results and recommendations
The second interim evaluation report confirms
the high value and significant achievements of the ARTEMIS and ENIAC JUs. In
particular, the panel concludes that (i) the relevance of the JUs remains high,
and considerable progress has been made to achieving their objectives; (ii) the
effectiveness is high; (iii) the efficiency is good despite the rather heavy
regulatory, administrative and financial burden; and (iv) the quality of reviewing,
reporting and monitoring of projects as done by the JU is high. Overall, the
panel is supportive to the tri-partite JTI instrument, pooling resources from industry,
the EU and
  Member States. Their recommendations call for further simplification,
improving administrative procedures and strengthening the governance
structures. The Commission welcomes and supports that the panel strongly
advocates the need for a single European strategy for Electronic Components and
Systems (ECS) research, development and innovation. This strategy should be developed
along with a matrix of Key European Industries (KEIs) to complement the Key
Enabling Technologies[8]
(KETs) as a means to identify, support and grow the industrial eco-systems
required for a sustainable European activity in ECS in a manner that is
compatible with the openness of the JU.
The detailed list of the panel's
recommendations is given in the annex. The timeframe for the recommendations
indicates whether they are to be implemented under the current JUs or should be
considered for the future JU, i.e. Electronic Components and Systems for European Leadership
(ECSEL) proposed by the Commission[9].
Regarding the implementation of the
recommendations of the first interim evaluation, the panel is overall satisfied
and indicates that most of the recommendations were addressed in the
Commission's report on the interim evaluation report and overall implemented in
a satisfactory manner. Nevertheless, the implementation of some
recommendations that were addressed to the Member States is according to the
panel less satisfactory. These recommendations relate to the
harmonisation of MS funding practices, procedures, and multi-annual budgetary
commitments which may cause process delays and administrative burden. 
5.            Planned follow-up actions
The
Commission welcomes the second interim evaluation report. The recommendations
are based on a thorough and critical analysis. The Commission will as set out
below implement the recommendations made to it and will work with industry, JTI member States and the Joint Undertakings to help them undertake their parts.
In
particular, the Commission will undertake:
·                        
short-term
actions for the existing JUs (i.e. their activities until the launch of the
new JTI) through their governance structures; and
·                        
actions in the context of the legal framework of the next generation JTI ECSEL.
The
Commission's response to the recommendations from the panel is outlined below.
This section also presents the follow-up by the Commission of its report about the
first interim evaluation.
5.1.         General observations
The Commission acknowledges the value of the tri-partite model and is
in particular pleased with the positive findings on the effectiveness and
efficiency of the current JUs. The launch and ramping up of the JUs has been
difficult but the major hurdles have been overcome.
In operational terms, the ENIAC JU is on track to bring it close to the
original ambition for the initiative, i.e. to leverage an industrial investment
programme in nanoelectronics R&D of some €2.7 billion. The ARTEMIS JU will be
further away from the initial target of €2.5 billion in R&D for embedded
computing systems. However, the developments of the last two years indicate
also for ARTEMIS a significant increase of its volume of activities. The
Commission is confident that both initiatives will have delivered on their
promises by the end of their mandate in 2017.
5.2.         Recommendations for the Industrial
Associations
The Commission recognises the important
role of the industrial associations being members of the JUs, especially in the
buy-in of industry including SME's, in keeping the research agenda up-to-date
and in promoting the JUs. The Commission stresses the importance of the
involvement of industry at the highest level to show leadership and engagement.
In this context, the positioning document[10] developed
under the auspices of AENEAS[11] and
CATRENE[12] provides significant perspectives and objectives for the
European nanoelectronics industry by 2020.
In agreement with the observations of the
second interim evaluation panel, the Commission considers the evolution of the
JUs to support projects at higher Technology Readiness Level (TRL) as both
timely and needed, putting more focus on innovation and exploitation, and bringing
R&D closer to the market. In particular, the call for manufacturing pilot
lines by the ENIAC JU in 2012 has been very successful. This call is the first
large-scale implementation of the recommendations from the High-Level Group on
KETs. Similarly, the ARTEMIS Innovation Pilot Projects are indeed 'a laudable
intent of sustaining innovation from proof of concept and prototyping stage
right through to a solid industrial platform'.
Furthermore, the good work done within
ARTEMIS-IA[13] on the portfolio analysis and key performance indicators
is acknowledged. Both industrial associations together with the JUs will be
encouraged to continue working in this direction thereby addressing
recommendation 15.
Nevertheless the Commission concurs with
the evaluation panel that more can be done as stated in recommendations 1 and
2. With the new ECSEL JU, the industrial partners should take the opportunity
of a more coordinated and proactive approach to reinforce the strategic
dimension of their cooperation. They will be invited to come up with key
orientations for the European electronic components and systems industry to
become the engine for the promising field of "smart everywhere"
products and services. To this effect, the proposal of the Council Regulation
on ECSEL includes a requirement for broader stakeholder engagement.
The report from the Commission on the
first interim evaluation commented extensively on the positioning of the JUs
towards EUREKA. The Commission now acknowledges that significant progress has
been made. In the area of nanoelectronics, CATRENE and ENIAC developed a common
Vision, Mission and Strategy document for R&D in micro- and nanoelectronics
in Europe. Similarly, in the area of embedded systems, ITEA2[14] and ARTEMIS hold co-summits and have set up a coordination
mechanism[15]. These actions will contribute to the development of an
overarching EU research, development and innovation strategy covering
nanoelectronics, embedded computing and cyber-physical systems in line with
recommendation 16 and should be reinforced. 
5.3.         Recommendations
for the Joint Undertakings
The Commission concurs with the findings of the panel
of experts that the evaluation and selection procedures and the technical
review process have been streamlined, are good and perceived by the
participants as being very helpful. Projects launched at the start of the JUs are
now coming to an end and the focus of the final reporting should be on the exploitation
of the results. The Commission therefore welcomes recommendations 3 and 15 and
will ask the Executive Directors to examine the possibility to implement them and
ensure a proper reporting on exploitation activities, though limiting the burden
on the beneficiaries. The development of an appropriate metrics for measuring
the impact and success of JU projects is equally important. 
As set out above, both JUs proved to be able to
respond quickly and appropriately to changing requirements and steer the
initiatives closer to the market. The various bodies in the JUs are to be
commended for this. Therefore, the Commission is confident that the
recommendations 4, 6 and 14 will find positive reception within the bodies of
the JUs as they address issues of a more operational nature.
The implementation of an ex-post audit strategy by
the JUs has been particularly difficult and subject to observations from the
Court of Auditors. The Commission will continue to raise this issue in the Governing
Boards in view of achieving a reasonable assurance that the financial
transactions of the JUs are correct. Recommendations 10, 12 and 13 are addressed
in the proposed Council Regulation on ECSEL to the extent in which these
recommendations can be implemented within the applicable legal framework.
5.4.         Recommendations
for the European Commission
The recommendations relating to the next generation JTI have been taken
into account in the Commission's proposal for a Council Regulation on the ECSEL
Joint Undertaking. This concerns in particular the recommendation to have a
single JTI (recommendation 5) with a single integrated research and innovation
agenda (recommendation 1) based on a simplified financial regulation
(recommendation 7) and an increased strategic role for its Governing Board
(recommendation 8). The ECSEL proposal furthermore incorporates a further harmonisation
of rules (recommendation 9) in accordance with the Rules for Participation of
HORIZON 2020.
Moreover the Commission adopted on 23 May 2013 a European strategy for
micro- and nanoelectronics components and systems[16], in line with
recommendation 16. A cornerstone of the strategy is the set-up of the new ECSEL
JU with a focus on innovation and covering higher TRL.
5.5.         Recommendations
for the Member States
The Commission recognises the difficulties
experienced by the JU member States in implementing their part of the recommendations,
in particular with respect to engaging in multi-annual commitments, the
harmonisation of rules and funding rates, and the synchronisation of procedures.
The experience and mutual understanding built up through their participation in
the ARTEMIS and ENIAC JUs should allow making a fresh start with the ECSEL JU under
the umbrella of Horizon 2020. This is already visible in the ECSEL Council Regulation
proposed by the Commission in which these issues are addressed. In particular,
it offers the possibility of a common approach for the implementation of the
public contributions to the projects. The Commission urges the Member States to
take all necessary measures to support the proposed harmonisation and simplification
of the processes for the beneficiaries in the future actions being retained for
funding.
The Commission also notes that much progress has been
made recently within ARTEMIS and ENIAC JU in finding a balance between supporting
national priorities and achieving common European objectives.
6.            Conclusion
While the first
interim evaluation helped the JUs improve their operations and focus on their
operational objectives, the findings of the second interim evaluation were a
timely and significant input in the drafting of the Commission proposal for the
future ECSEL Council Regulation. Although there are still administrative
hurdles which can be overcome in the future ECSEL JU, the ARTEMIS and ENIAC JUs
proved to be an effective and efficient tool to implement an R&D programme driven
by industry and co-financed by the Member States and the EU.
The final evaluation
of the ARTEMIS and ENIAC JUs is scheduled for 2017 as part of the interim
evaluation foreseen for the ECSEL JU. That final evaluation should look into
the overall impact of the projects supported by the JUs, and comment on their
contribution in the overall context of the European strategy for micro- and
nanoelectronics and embedded systems.
 Annex - List of recommendations by the
evaluation panel
 No[17]   || Summary of recommendations || Timeframe 
 Recommendations for the Industrial Associations 
 1 || The JTI SRA and work programmes need to reflect more strongly a coherent European perspective, linking to an overarching European Electronic Components and Systems research, development and innovation strategy, as proposed in Recommendation 16. || Next Generation JTIs 
 2 || The Industrial Associations should play a more active role in the definition of the overall objectives and strategy of the JTIs and should engage more actively with stakeholders so as to promote and facilitate participation in project proposals, especially by SMEs, and to develop and keep up-to-date the Strategic Research Agenda. || Now/ Next Generation JTIs 
 Recommendations for the Joint Undertakings[18] 
 3 || JTI project reviews, including a final post-project review that should be held, the panel concludes, between 6 and 12 months after the end of a project, should monitor more closely and rigorously the actual and planned exploitation of project results, and the measures put in place by project partners to achieve such planned exploitation. || Now 
 4 || ARTEMIS projects should build, where appropriate, on previously developed ARTEMIS technology, making reference to what has been funded before and demonstrating, in addition to novelty, the appropriate re-use of previous project results combined with a suitable progression to higher TRL levels. The proportion of funding for projects targeting generic applications and services (Applications projects) should be increased. || Now/ Next Generation JTIs 
 6 || ENIAC and CATRENE calls for, and selection of, proposals should be more closely aligned (e.g. by the use of common and/or complementary calls), with the relevant funding awarding bodies retaining some flexibility over the assignment of the most appropriate funding stream. || Now/ Next Generation JTIs 
 8 || Focus the JU Governing Boards on strategic issues and reduce their administrative burden in order to attract participation from high-level industry representatives. || Now/ Next Generation JTIs 
 10 || The JUs should explore and develop appropriate mechanisms to create an ‘early warning system’ to identify potential delays, or restrictions to the availability, of funding from Member States. In order to bridge any financial gaps so identified, advanced funding by the EC (on behalf of a Member State) should be allowed for projects which are mission-critical. || Now 
 12 || Take steps (e.g. modification of evaluation criteria) during the proposal evaluation and selection process to improve the match of the project portfolio to strategic European aims and to ensure optimum coverage of key areas defined in the overarching EU ECS strategy (proposed in Recommendation 16) and the workplans derived from such a strategy. || Next Generation JTIs 
 13 || Specific support mechanisms for enhancing the project management processes in JTI projects should be developed and implemented. Management costs should be 100% funded by the EC for all JTI projects. || Now/ Next Generation JTIs 
 14 || JTI projects should be subject to only one (i.e. the JU) project review and reporting process. || Now 
 15 || Appropriate metrics for measuring the impact and success of JTI projects should be developed and used for both current and future JTIs. || Now 
 Recommendations for the European Commission 
 5 || The ENIAC & ARTEMIS JTIs, along with the European Technology Platform (ETP) on Smart Systems Integration (EPoSS), should be integrated into a single organisation (an ECS JTI). || Next Generation JTIs 
 7 || Construct the proposed new, integrated JTI (of Recommendation 5), or indeed any future JTI, as a PPP body as defined in Article 209 of the financial regulation. || Next Generation JTIs 
 13 || Specific support mechanisms for enhancing the project management processes in JTI projects should be developed and implemented. Management costs should be 100% funded by the EC for all JTI projects. || Now/ Next Generation JTIs 
 16 || A mid- to long-term overarching EU research, development and innovation strategy in Electronic Components and Systems (ECS) should be clearly defined and used as a key 'driver' for funding decisions. || Now 
 Recommendations for the Member States 
 9 || Member State participation rules, funding rates and procedures should be harmonised and synchronised wherever possible, adopting best practice as the guiding principle. || Now/ Next Generation JTIs 
 11 || Member States should commit to a multi-annual funding system. || Now/ Next Generation JTIs 
 14 || JTI projects should be subject to only one (i.e. the JU) project review and reporting process. || Now 
[1]               Council Regulation No 74/2008 of 20 December 2007 on
the establishment of the ‘ARTEMIS Joint Undertaking’ to implement a Joint
Technology Initiative in Embedded Computing Systems.
Council Regulation No 72/2008 of 20 December 2007 setting up the ENIAC Joint
Undertaking.
[2]               ARTEMIS member States: Austria, Belgium, Cyprus
(withdrew recently), Czech Republic, Denmark, Estonia, Finland, France,
Germany, Greece, Hungary, Ireland, Italy, Latvia, Netherlands, Norway, Poland,
Portugal, Romania, Spain, Sweden, Slovenia, United Kingdom.
ENIAC member States: Austria, Belgium, Czech Republic, Estonia, Finland,
France, Germany, Greece, Hungary, Ireland, Italy, Latvia, Malta, Netherlands,
Norway, Poland, Portugal, Romania, Slovak Republic, Spain, Sweden, United
Kingdom.
[3]               http://ec.europa.eu/dgs/information_society/evaluation/rtd/jti/index_en.htm
[4]               COM(2010)
752 final and http://ec.europa.eu/dgs/information_society/evaluation/rtd/jti/
[5]               The expert panel consisted of the following members:
M. Götzeler (CEO Aixtron SE); W. Arden (former Infineon Technologies); Ch. de
Prost (ATMEL); J.-L. Dormoy (EDF); M. Jansz (Technology Foundation STW); T.
Luukkonen (Research Institute of the Finnish Economy); A.
Sangiovanni-Vincentelli (University of California at Berkeley); D. Wright (University of Exeter).
[6]               The evidence base for the evaluation included an
extensive desk review of relevant documents on legal and financial matters,
research agendas, work programmes, participation statistics and project
information. 104 interviews were conducted with a wide range of stakeholders:
industry, academia, Commission, national Public Authorities, EUREKA clusters
and the JUs.
[7]               https://ec.europa.eu/digital-agenda/en/news/second-interim-evaluation-artemis-and-eniac-joint-technology-initiatives
[8]               http://ec.europa.eu/enterprise/sectors/ict/files/kets/hlg_report_final_en.pdf
[9]               COM(2013) 501 final and http://ec.europa.eu/research/press/2013/pdf/jti/ecsel_factsheet.pdf
[10]             Innovation for the future of Europe: Nanoelectronics
beyond 2020           
http://www.aeneas-office.eu/web/downloads/strategic-docs/position_paper_final.pdf
[11]             The Association for European NanoElectronics ActivitieS         
http://www.aeneas-office.eu/web/index.php
[12]             The Eureka Cluster for Application and Technology Research
in Europe on NanoElectronics
http://www.catrene.org/
[13]             The industrial Association for Advanced Research &
Technology for EMbedded Intelligence and Systems   
http://www.artemis-ia.eu/
[14]             The Eureka Cluster on Information Technology for
European Advancement (ITEA)             
http://www.itea2.org/
[15]             High-level vision 2030 ITEA-ARTEMIS - http://www.artemis-ia.eu/publications
[16]             COM(2013) 298 final
[17]             The numbering of the recommendations refers to the
report of the experts.
[18]             These recommendations refer to a collective/joint
responsibility of all parties involved in the JUs.