EUROPEAN COMMISSION Brussels, 19.1.2016 SWD(2016) 2 final Ex-Post Evaluation of the Seventh Framework Programme COMMISSION STAFF WORKING DOCUMENT Annexes Accompanying the document COMMUNICATION FROM THE COMMMISSION TO THE COUNCIL, THE EUROPEAN PARLIAMENT, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS

On the response to the Report to the High Level Expert Group on the Ex-Post Evaluation of the Seventh Framework Programme {COM(2016) 5 final}

{SWD(2016) 1 final}



COMMISSION STAFF WORKING DOCUMENT Annexes Accompanying the document COMMUNICATION FROM THE COMMMISSION TO THE COUNCIL, THE EUROPEAN PARLIAMENT, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS

On the response to the Report to the High Level Expert Group on the Ex-Post Evaluation of the Seventh Framework Programme Contents 1.Procedural information concerning the process to prepare the evaluation 2.Stakeholder consultation 3.Methods and Analytical models used in preparing the evaluation 4.List of evaluation studies 5. Evaluation questions 6.Background to FP7 7.Overview of FP7 indicators 8.Background on current situation 9.The Cooperation Specific Programme 10.Thematic areas of the Cooperation Programme





1.Procedural information concerning the process to prepare the evaluation Lead DG: Directorate General Research and Innovation (RTD) Agenda Planning number: 2015/RTD/004 Commission Communication on the ex-post evaluations of the EC and Euratom 7th Framework Programmes The Ex-Post evaluations of the EC and Euratom Seventh Framework Programmes started in 2014. The main justification derives from Article 7 paragraph 3 of Decision 1982/2006/EC setting up FP7 which stipulates that "two years following the completion of this Framework Programme, the Commission shall carry out an external evaluation by independent experts of its rationale, implementation and achievements”. At the first stage the Inter-Service RTD Evaluation Network functioned as the Inter-Service Group (ISG) for this evaluation. After the adoption of the Better Regulation Guidelines a specific ISG was set up in June 2015 specifically for the preparation of the Staff Working Document on FP7 Ex-Post Evaluation. This ISG met three times before the submission of the Staff Working Document to the Regulatory Scrutiny Board (30 June, 29 July and 3 September 2015). In accordance with the feedback received from the Regulatory Scrutiny Board, the Staff Working Document has been revised in the following ways. The qualitative assessment of the different aspects of FP7 has been strengthened. This has included greater explanation of the reasons why certain aspects of FP7 were particularly successful while others worked less well. In this regard, a more detailed assessment of the impact of simplification measures introduced in FP7 has been included. Greater attention has also been paid to setting FP7 in the wide global context and its contribution to the EU’s objective of becoming the world’s leading research area. Further analysis has also been added as regards the coherence between FP7 and other related programmes, including the Cohesion Funds. Metholodogical and presentational aspects have been clarified. The intervention logic has been made more transparent. Graphical presentation of data has been streamlined to make FP7 results and outcomes more comparable. The evaluation studies used for this evaluation are listed in Annex 4 and the results of the over 150 evaluation studies are described in Annexes 9-22. Majority of the evaluation studies were carried out by external experts and contractors.



2.Stakeholder consultation Introduction The Ex-Post Evaluation of the 7th Frameworks Programme for Research (FP7), required also by the legal base, is an important instrument for informing the European Parliament and the Council, Member States, the research community, the general public and other stakeholders about the achievements of FP7. It will also contribute to improving implementation of Horizon 2020 and provide a solid evidence base for designing future framework programmes. The evaluation will examine the effectiveness of FP7 implementation, the efficiency of resources used and the wider socio-economic impacts of the Framework Programme. In order to provide the evaluation with a range of opinions and views about the functioning, achievements, and impacts of FP7, a public consultation ran from February to May 2015 allowing for contributions both from those with direct experience with the FP7, as well as groups or individuals who wished to give their views. This is an overall summary of the contributions to the consultation. 2.1. Executive summary In general FP7 was well received: The overall satisfaction with FP7 was high (80 %) among those who participated in the consultation. The key strengths of the programme can be summarised as follows:

-Creating the European Research Area through cooperation and competition: Networking people and organisations over the geographic, sectorial and disciplinary borders (international, inter-sectorial and interdisciplinary collaboration) and funding the highest quality research through competition -Joint Agenda-setting: Joining forces in solving the grand challenges together (e.g. through JTIs, PPPs, ETPs) -European Research Council (ERC)/IDEAS Specific Programme was a great success -Marie Curie Actions/PEOPLE Specific Programme made an important impact for the future of research through training and creating opportunities for mobility At the same time there were some shortcomings: -Administrative burden was high and financial and legal rules were cumbersome -Societal impact was not addressed to a sufficient extent -Perception of sometimes narrow topics and of difficulty to enter for newcomers: Game of big entities who know how the system works -Regarding the impact of the Programme, according to the respondents, the biggest impact of FP7 was on scientific excellence and on technological and social innovations. According to the input received, among the ERA priorities, FP7 contributed most to i) Optimal transnational co-operation and competition, ii) Optimal circulation, access to and transfer of scientific knowledge, iii) Open labour market for researchers. More specifically for SMEs the ERA priority to which FP7 contributed the most was "Open labour market for researchers". At the overall level EU added value of FP7 was demonstrated through: i) Tackling pan-European challenges, ii) Increased competition in research, iii) Enhance researchers’ mobility. For individuals the most important area of EU added value was the enhancement of researchers' mobility whereas for ministries and agencies the most important areas were the improvement of S&T capabilities and the increased competition in research. The simplification measures taken were well received, and the need for further simplification was raised by a number of respondents. 2.2.1 EECS/RTD Event "Have your say on FP7" On 27 October 2015 from 14:30-18:00 the EESC together with RTD held an event on the public consultation. About 120 people participated. The purpose of the conference was to present: -The FP7 Ex Post Evaluation set-up -The findings of the online consultation on FP7 -Expert opinions on scientific impact -Expert opinions on impact on innovation -Expert opinions on socio-economi impact

-Expert opinions on European added value -Key achievements and shortcomings of FP7 All the presentations can be found on this link . The presentations were followed by a moderated discussion allowing participants to air their thoughts and views. The main topics for discussions were: Simplification, open access and industry collaboration, discussion on scope, civil society's role and Technology Readiness Levels.

2.2. Context of the public consultation on FP7 Framework Programmes The Framework Programmes are the EU’s main instruments for the funding of research and innovation in Europe. Based on the Treaty establishing the EU , the Framework Programmes serve two main strategic objectives: strengthening the scientific and technological bases of industry and encouraging its international competitiveness while promoting research activities in support of other EU policies. The 7th Framework Programme (FP7) pursued the general objectives described in the Treaty to strengthen industrial competitiveness and to meet the research needs of other Community policies, thereby contributing to the creation of a knowledge-based society, building on a European Research Area and complementing activities at a national and regional level. It promoted excellence in scientific and technological research, development and demonstration through four specific programmes: Cooperation, Ideas, People and Capacities. Horizon 2020 is the on-going EU Research and Innovation programme with nearly €80 billion of funding available over 7 years (2014 to 2020). Horizon 2020 aims to contribute to building a society and an economy based on knowledge and innovation across the Union by leveraging additional research, development and innovation funding and by contributing to attaining research and development targets. Horizon 2020 supports the implementation of the Europe 2020 strategy and other Union policies, as well as the achievement and functioning of the European Research Area (ERA). Evaluation framework The Decision setting up FP7 stipulates that "two years following the completion of this Framework Programme, the Commission shall carry out an external evaluation by independent experts of its rationale, implementation and achievements." The evaluation is an important instrument for informing the European Parliament and the Council, Member States, the research community, the general public and other stakeholders about the achievements of FP7. It also contributes to improving implementation of the current Framework Programme, Horizon 2020 and provides a solid evidence base for designing future framework programmes. The evaluation examines the effectiveness of FP7 implementation, the efficiency of resources used and the wider socio-economic impacts of the Framework Programme. The evaluation covers the entire period of FP7 implementation in between 2007-2013. In order to provide the evaluation with a range of opinions and views about the functioning, achievements, and impacts of FP7, the public online consultation was set up to allow for contributions both from those with direct experience with the FP7, as well as any groups or individuals who wished to give their views. The questionnaire is attached as Annex 1. Responses 202 responses were encoded to the public online consultation between February and May 2015. The responses came from 24 EU member states (all except for EL, HU, LV and SK) and additionally 6 responses were sent from other countries (such as Switzerland and Turkey). The respondents were asked whether they respond as individuals or whether they represent an organisation. The results are presented separately for overall, individual respondents and for different types of organisations. -88 (44%) higher education and public research organisations (HES & RES) -70 (35%) individuals -20 (14%) private sector (PRC)

-17 (8,4%) ministries and agencies (PUB) -7 (3,5%) SMEs Most of the questions in the questionnaire were compulsory single/multiple choice questions. In addition the respondents were provided with an option to comment on each theme with an optional open reply. In addition to the responses to the online questionnaire 10 organisations provided their individual written contributions separately. These contributions are referred to at each relevant section of this analysis (Separate written contributions). 2.3. Key figures on the public consultation and main outcomes Overall satisfaction and effectiveness of implementation At the overall level 68 % of the respondents were satisfied or very satisfied with FP7 whereas 17 % were moderately or very dissatisfied. Only a handful of respondents indicated that they were very dissatisfied. Ministries and funding agencies responded "Very satisfied" most often, whereas individual respondents tended to respond "Moderately or very dissatisfied" more often than other groups of respondents. SMEs were more likely to reply "Very satisfied" or "Very dissatisfied" compared to other respondents. Figure 1. Overall are you satisfied with FP7? The large majority of the responses (92 %) agreed that the implementation of FP7 was effective in general whereas only 8 % considered that the implementation has not been effective. Figure 2. Based on your experience has the implementation of FP7 been effective? In the comments received a number of respondents, both public and private bodies, noted that the incoherence in the implementation of rules and regulations influenced the effective implementation of the Programme negatively. According to LERU (League of European Research Universities) "the vast majority of the problems faced were related to the administrative overhead and red tape they were confronted with when participating in, and especially when coordinating, an FP7 project."

Separate written contributions According to the contribution from the European University Association (EUA) FP7 was a key driver of enhanced European university research and innovation development and cooperation. Also, according to the UK higher education institutions the impact of European research depends crucially on international collaboration and the mobility of researchers across Europe. FP7 was vital in enabling that collaboration and mobility. The Swiss National Science Foundation (SNSF) reported that in general the implementation of FP7 was mostly successful. On the other hand, in its separate input Science Europe, UK Department for BIS and Research Councils UK (RCUK) call for more transparent governance (including consistency, coherence and transparency) and for a more balanced mix of top-down and bottom-up approaches. Key achievements and strengths of FP7 Question: What are the key achievements/strengths of FP7 in particular? Key achievements reported by the respondents: -Creating the European Research Area through cooperation and competition: Networking people and organisations over the geographic, sectorial and disciplinary borders (international, inter-sectorial and interdisciplinary collaboration) and funding the highest quality research through competition -Joint Agenda-setting: Joining forces in solving the grand challenges together (e.g. through JTIs, PPPs, ETPs) -European Research Council (ERC)/IDEAS Specific Programme was a great success -Marie Curie Actions/PEOPLE Specific Programme made an important impact for the future of research through training and creating opportunities for mobility -SMEs highlighted especially the networking effect of FP7 and efforts made to involve all the different parts of the value chains. For the private companies, one of the key achievements seemed to be the JTIs setting the agenda jointly for the public and private actors. According to HES and REC the key achievements of the Programme were investing firstly in collaborative, transnational research and networking and secondly in scientific excellence through ERC. For Ministries and Agencies the main achievements of FP7 were the input to scientific excellence, notably through IDEAS Specific Programme and to research careers and mobility. Other observations on achievements by several respondents: -Level of funding was good (especially compared to the national R&I funding) -The efforts made to simplify were welcome (e.g. Research Participant Portal) although not sufficient -The efforts made to widen the access to major research infrastructures were appreciated Shortcomings Question: Are there shortcomings in FP7 that you think should be corrected? According to your experience have these already been addressed to in the Horizon 2020 Programme? Main issues reported by the respondents -Administrative burden was high and financial and legal rules were cumbersome -High oversubscription -Societal impact was not addressed in calls and/or projects to a sufficient extent

-Too narrow topics and too limited call scopes -Industry participation was neglected: FP7 did not enhance industry participation enough -Difficult to enter for newcomers: Game of big entities who know how the system works, lack of transparency -FP7 was fragmented and inconsistent: Need for Commission and Agencies internal coherence (e.g. different interpretations of rules) While some of the respondents representing industry claimed that the Programme was directed towards the needs of the academia, at the same time some respondents from the HES and REC side considered that public funding should not be channelled to respond to the interests of the private companies. But overall the respondents were quite unanimous irrespective of their background (private or public, SME or individual). Also, it was noted repeatedly that many of the shortcomings of FP7 have been corrected in Horizon 2020. As one individual respondent put it: "Many shortcomings were addressed in H2020, namely: faster time to grant, more possibilities for open topic calls, decreasing of the fragmentation by programmes, types of instruments etc." Separate written contributions SNSF highlighted the following issues as shortcomings: Fragmentation, regulatory and administrative issues (lack of clarity and consistency) The endurance of FP7 impact At the overall level nearly 90 % of the respondents considered that FP7 research activities produced enduring impact to high or medium extent. Less than 10 % of the respondents perceived that FP7 produced enduring impact only to low extent. Private organisations (other than SMEs) were more likely to consider that FP7 activities produced enduring impact to a high extent. Individual respondents and SMEs were more likely than others to respond that FP7 produced enduring impact only to a low extent. Figure 3. Based on your experience to what extent did FP7 research activities produce enduring impact? Regarding the question "To what extent did FP7 research activities produce enduring impact for you as FP7 beneficiary (e.g. networking, benchmarking, joint agenda setting and harmonisation of peer review systems)?" for the individual respondents and for the SMEs the impact on networking was clearly the most important. For the HES and REC the comments concerned the FP7 having a great impact in increasing the collaborations in research projects, partnerships and networking. According to the private companies the impact of FP7 is based on the joint agenda setting and strategic cooperation. For the Ministries and Agencies, the biggest influence of FP7 was on the contribution to the peer learning and exchange of good practices in addition to the networking effect. Separate written contributions According to EUA FP7 facilitated scientific and technological cooperation across European universities which will have a lasting effect with respect to academic research staff and young researchers exchange and career development. RCUK believes that funding for transnational pan-European collaborative research lies at the heart of the Framework Programme.

Impact of Specific Programmes The respondents were asked to assess, for each of the four Specific Programmes of FP7, the areas to which that specific programme created most impact . For all the four Specific Programmes the respondents perceived that the impact of Specific Programmes on scientific excellence was the greatest followed by the impact on technological and social innovations and the economic impact. For the People Specific Programme the Societal impact was considered more important than the economic impact. For Capacities Specific Programme the regional and societal impacts were considered more important than for other Specific Programmes. Regarding the views of different respondent groups, interestingly societal impact was considered more important by individuals and HES & REC than by other respondent groups.

Figure 4. In which of the following areas did COOPERATION Specific Programme of FP7 generate most impact? Figure 5. In which of the following areas did IDEAS Specific Programme of FP7 generate most impact? Figure 6. In which of the following areas did PEOPLE Specific Programme of FP7 generate most impact? Figure 7. In which of the following areas did CAPACITIES Specific Programme of FP7 generate most impact? In the optional open comments the respondents highlighted the underlining objectives of FP7 to support excellent science and technological and social innovations thus stating that FP7 was achieving the objectives set in these terms. At the same time some respondents made the remark that for the evaluation of the societal or economic impact of FP7 an in-depth analysis would be required. Overall, according to the input received the four different Specific Programmes seem to cover the different needs of the complex research and innovation system and thus seem to generate impact as expected. As one respondent from the private side put it: "In their impact, the 4 Specific Programmes were complementary". Separate written contributions

According to the contribution by Science Europe “excellence fosters excellence”, thus the Framework Programmes must remain excellence-based programmes. Similarly, RCUK recognised the added value of the EU funding for frontier research, mobility programmes, collaborative research and research infrastructures. According to UK Department for BIS FP7 provided significant added value to national research activities and in many areas helped to establish critical mass beyond the reach of national capability but it is far too soon to fully assess the programme’s economic impact. This was also backed up by the contribution of UK HE institutions.

The impact of simplification measures In the course of FP7 major simplification measures were introduced to reduce the burden to the participants. The respondents were asked to assess the impact of various simplification measures . At the overall level “Research Participant Portal” was considered to have been the measure having most impact on simplification. This is clearly the view of the individual respondents and the representatives of HES&REC. SMEs considered that the flat rate system for SME owners and the wider acceptance of average personnel costs were the most efficient simplification measures. For the other private sector organisations the simplification measures having most impact were the unique Registration Facility (URF) and the web-based electronic system for negotiations (NEF). Figure 8. Which of the following FP7 simplification measures generated most impact? In the replies to the optional open question on simplification measures the views of the respondents were divided. On one hand a number of respondents highlighted their experiences on the administrative burden, notably on the usefulness of the reporting requirements. On the other, many respondents underlined the success of the simplification measures taken, especially the Participant Portal and the Electronic System for negotiations. As one respondent put it: "The simplification measures succeeded in making the implementation of the FP7 projects manageable and smooth. The Research Participant Portal has been the most useful tool for the negotiation, implementation and reporting." Separate written contributions According to LERU the FP7 simplification measures were "reasonably successful". Furthermore LERU stated that "the level of detail requested at reporting stage has increased significantly throughout FP7 turning it into a laborious exercise." According to the contribution of EUA “the overall effectiveness could have been enhanced more with greater simplification of the procedures and regulations governing the grants and contracts”. In detail, EUA asks for further simplification of rules and regulations, procedures and cost recovery. According to Science Europe the complexity of funding instruments made the access to FP7 more difficult to newcomers and the agenda-setting less transparent. UK Department for BIS reported that no single simplification measure had a major impact but the collective impact was welcome especially because the programme was still regarded as bureaucratic and complex. According to RCUK simplification measures were a welcome step towards reducing the administrative burden. Contribution of FP7 to European Research Area priorities The respondents were asked to assess to which of the European Research Area (ERA) priorities FP7 contributed the most . At the overall level the clear majority considered that FP7 contributed most to the ERA priority “Optimal transnational co-operation and competition”. This refers also to the specific section on key achievements of FP7 (see section 5.2.).The importance of the contribution by FP7 to the ERA priorities ranked according to the responses:



1. Optimal transnational co-operation and competition 2. Optimal circulation, access to and transfer of scientific knowledge 3. Open labour market for researchers 4. More effective national research systems 5. Gender equality and gender mainstreaming in research For SMEs the ERA priority to which FP7 contributed the most was "Open labour market for researchers". Concerning the views of HES and REC, the biggest contribution from FP7 was to the "Optimal transnational cooperation and competition". One respondent from HES found: "The more funding available for collaborative research the more ERA will become reality, through researchers working together, moving to work etc. Large-scale International projects (enabled by FP7 & H2020) are essential." Moreover, reflecting the importance of the open labour market perceived by HES and REC, they also underlined the importance of continuing the efforts to improve researcher training, career perspectives and mobility to achieve ERA. The Ministries and Agencies assessed FP7 having contributed to the more effective national research systems more often than other groups of respondents. In the written comments, one respondent noted: "The Joint Programming initiatives have increased the effectiveness of research and reduced duplication of efforts by strategically aligning the research funding of member states, implementing joint research agendas and jointly investing in research infrastructures. Secondly, the FP7 has facilitated the spreading of practices such as peer review, competitive funding, open access, transparent recruitment, gender mainstreaming etc." An interesting aspect is that in the multiple choices question Ministries and Agencies considered that FP7 did not contribute at all the ERA priority "Gender equality and gender mainstreaming in research". Figure 9. Contribution of FP7 activities to the European Research Area (ERA)

Separate written contributions

According to EUA, “FP7 has been instrumental in supporting universities in consolidating their actions towards the ERA goals”. Among the key aspects in these terms according to EUA were adequate public funding mix for university activities, nurturing of the open labour market for researchers (especially with respect to university-business collaboration and mobility) and the enhancement of optimal circulation, access to and transfer of scientific knowledge. The shift towards open access and open science in general during FP7 was welcome according to several individual contributions (e.g. UK BIS and Science Europe). Moreover, the ERA-NETs and joint programming were highlighted as schemes offering added value by networking the funding the agencies across the Europe. Furthermore, according to RCUK among others, ERA benefitted from FP7 Research Infrastructure programme. Success of FP7 in achieving EU added value The respondents were asked to assess at which of the areas of the EU added value FP7 was most successful . The three most important areas of EU added value where FP7 was most successful ranked according to the respondents: 1. Tackling pan-European challenges 2. Increase competition in research 3. Enhance researchers’ mobility For individuals the most important area of EU added value was the enhancement of researchers' mobility whereas for Ministries and Agencies the most important areas were the improvement of S&T capabilities and the increased competition in research. From the perspective of the private bodies, the added value of FP7 was specifically channelled through tackling the pan-European challenges and through reducing the risks. In the optional open comment one representative from the private side noted that: "By bringing additional public support and fostering critical mass for ambitious but risky research programmes, FP7 has addressed market failures and reduction of technical and business risks."

Figure 10. In which of the following dimensions of EU added-value has FP7 been most successful? Separate written contributions In the view of EUA the EU added value of FP7 is illustrated around European Research Council (ERC), Marie Curie Actions and European Institute of Innovation and Technology (EIT). In a similar way, in their inputs, Science Europe and UK Department for BIS considered the ERC and the instruments for researcher mobility as the key strengths of FP7. RCUK considers FP7 successful in terms of funding for EU added value activities and the opportunities for collaboration and competition on a pan-EU scale. Results to marketable products At the overall level around 1/3 of the respondents estimated that the results of their research will lead to marketable products and services within 1-5 years and ¼ considered that the same will happen within 5-10 years. A large share of SMEs and the private organisations tend to expect the results of their project to lead to marketable products and services already within one year. However, up to 1/3 of the respondents chose the alternative “Don’t know” to this question. This can be explained by the large share of “non-participants” among the respondents (e.g. the Ministries and funding agencies).

Figure 11. How quick do you think the result of your research will lead to marketable products and services? 2.4. Further remarks On participation and non-participation Almost 2/3 of the respondents knew theirs partners in the project beforehand and only ¼ got to know their partners only at the start of project. Nearly 90 % of the respondents intend to stay in touch with their project partners also in the future. Less than 20 % replied that they participated for the first time. 92 % of the respondents also intend to participate in the Framework Programmes in the future.

An optional open question was posed for the non-participants on the reasons for not participating ("In case you have not been participating FP7, please specify the reasons for non-participation"). The main remarks received commented on the cumbersomeness of big consortia (e.g. finding international partners) and the perception of consortia being "closed clubs for those who already know each other". This view was reported by both some individual respondents and some public and private bodies. Separate written contributions Important aspects to be further addressed during Horizon 2020 are according to EUA the following: The continuation of bottom-up funding instruments and the coverage of the whole value chain by instruments Enhancing the integral role of the social sciences, arts and humanities Strengthening the international dimension of Horizon 2020 and coordinating regional/national/European R&D and innovation programmes Wider interpretation of innovation and further promotion of knowledge partnerships (links between education, research and business) Science Europe calls for mid- to long-term vision and commitment for excellence research and finds it worrying that the research agenda is more and more industry-led and subject to short-term goals. Furthermore they emphasize the importance of an instrument portfolio enabling international cooperation and third country participation. Science Europe Scientific Committees of Social Sciences and Humanities provided an input covering specific issues related to the participation of social sciences and humanities in the Framework Programmes. According to this the key achievements of FP7 were the community building, contribution to the growing scientific ambitions, the support for global competition and the investment in infrastructures. As shortcomings of FP7 the Science Europe SSH Committees highlight the project design format more appropriate to industrial processes than for SSH research, the under-estimation of SSH research contribution and the focus on large-scale and rigid linear organisation of the research projects. 2.5. Conclusions As stated in the Better Regulation Guidelines the design, evaluation and revision of policy interventions benefit from considering the input and views provided by stakeholders. Consultations can also improve the evidence-base of the evaluation. This was clearly the case also for the Ex-Post Evaluation of FP7. According to the input to the public consultation on the FP7 Ex-Post Evaluation the overall satisfaction with the Framework Programme was good. The respondents of this consultation state clearly that FP7 was creating EU added value, mainly by tackling pan-European challenges, by increasing competition in research and by enhancing researchers’ mobility. Strong support is given to boosting excellence in fundamental research, notably through European Research Council. The respondents find that FP7 contributed to the creation of European Research Area, notably by supporting the optimal transnational cooperation and competition, the optimal circulation, access to and transfer of knowledge and the open labour market for researchers. According to the responses there seems to be clear support for increasing the excellence of European science through policies that reinforce openness and integrity of science. The results also suggest that reinforcing international engagement is needed. Based on the respondents' perceptions on whether the FP7 has an enduring impact and how quickly the results would lead to marketable products the FP7 made a clear effort to capitalise on the results of research and to create a vibrant innovation ecosystem. Where there was room for improvement according to the respondents a lot has already been done for Horizon 2020. One of the main shortcomings of FP7 was the heavy administrative burden. Here, the efforts made and the measures taken to simplify were welcomed and well received.



2.6. The online questionnaire Public online stakeholder consultation on the Ex-Post Evaluation of FP7 I. Context and background The Decision setting up 'Seventh Framework Programme’ (FP7) of the European Community for research, technological development and demonstration activities' stipulates that "two years following the completion of this Framework Programme, the Commission shall carry out an external evaluation by independent experts of its rationale, implementation and achievements." The evaluation is an important instrument for informing the European Parliament and the Council, Member States, the research community, the general public and other stakeholders about the achievements of FP7. It will also contribute to improving implementation of Horizon 2020 and provide a solid evidence base for designing future framework programmes. The evaluation will examine the effectiveness of FP7 implementation, the efficiency of resources used and the wider socio-economic impacts of the Framework Programme. The evaluation covers the entire period of FP7 implementation in between 2007-2013. The evaluation will cover the objectives of FP7 as set out at the time when the objectives originally were set (2005-2007). At the same time, it should take into account that the context has changed significantly during the period of programme implementation. Several developments (in the context of FP7 or with a significant impact on the programme) influenced the evolution of FP7 over this period: The size of the EU Budget allocation to the research activities of FP7 grew substantially both in real terms and as a proportion of the overall budget;

New initiatives to stimulate the European Research Area were launched; The European Research Council (ERC) was created; A range of new activities and implementation schemes were introduced during FP7. Moreover, in the light of the financial and economic crisis, research efforts have been more than ever expected to help address major challenges. Besides these external factors, entry into force of the Lisbon Treaty in 2009 and the Europe 2020 strategy have extended the scope of European research policy and positioned it as a key component of growth and competitiveness, together with innovation. Furthermore, over the life of FP7 there has been an increased focus on accountability and the need to demonstrate more concretely what impact has been achieved with the resources devoted to the Framework Programme. This aspect was raised in the ex post evaluation of the Sixth Framework Programme (2009) and the Interim evaluation of the Seventh Framework Programme (2010). The ex-post evaluation covers research programme activities under FP7, involving almost 25,000 research projects and signed grant agreements in four specific programmes - Cooperation, Capacities, Ideas and People . Currently around 50 % of the projects are still running. The ex-post evaluation is carried out by an independent High Level Expert Group, supported by an extensive evidence-base. It will be completed by the end of 2015 as is required by the legal basis . This will be followed by a Commission Communication on the Ex-Post Evaluation. In order to provide the experts with a range of opinion and views about the functioning, achievements, and impacts of FP7, this interactive consultation has been set up to allow for contributions both from those with direct experience with the FP7, as well as groups or individuals who wish to give their views. The results of this consultation will be made publicly available and will be taken into account in the Commission Communication reacting to the Evaluation Report of the High Level Expert Group. II. List of questions 1. Information about the respondent (compulsory) 1.1 Do you reply As an individual On behalf of an organisation 1.2 Your role in the organisation -None – I am answering as an individual -Senior management

-Management -Researcher -Strategy /policy function -Specialist/Expert -Other (please specify) 1.3 Country of origin (of the organisation when relevant) [to choose from a list]

1.4 Your organisation's geographical area of activities (indicate your area of activities if answering as an individual person): -Local -Regional -National -European -International -Not applicable 1.5 Your organisation's type of activity (indicate your activity type if answering as an individual person) -Higher or Secondary Education (HES) -Research Organisation (REC) -Private For-Profit (excluding education) (PRC)

-If SME, please indicate -Public body (excluding research and education) (PUB) -Research and/or Innovation Funding Agency -Ministry -Other -Other (i.e. NGO, association, please specify) 2. Implementation of FP7 2.1 Based on your experience has the implementation of FP7 been effective? (compulsory) With implementation we refer to the overall management of the framework programme, i.e. communication on calls, application and grant negotiation procedures and dissemination of project findings. -Yes -Generally yes, but with some problems -Generally no, although with some successes -No -Don't know 2.2 Has the implementation of FP7 been effective? - Comments (optional) Please specify the reasons for the reply given under 2.1.

2.3 In case you have not been participating FP7, please specify here the reasons for non-participation (optional) 3. Simplification of FP7 3.1 Which of the following FP7 simplification measures generated most impact? (compulsory) Out of the 11 FP7 simplification measures listed below, please select the 5 FP7 simplification measures which, in your view, generated most impact and rank them accordingly (5 generating most impact)) -Certification of costs (fewer audit certificates) -Participants Guarantee Fund (fewer ex-ante financial checks) -Unique Registration Facility (URF) -Certification of methodology -Web-based electronic system for negotiations (NEF) -Project reporting – streamlined guidelines and structure of reports -Grant amendments – streamlined rules and procedures -Research Participant portal -Simplification of recovery process (flat rate corrections) -Wider acceptance of average personnel costs -Flat rate system for SME owners and natural persons without salary

3.2 To what extend have the FP7 simplification measures been successful? - Comments (optional) 4. Achievements and impact 4.1 Impacts of each Specific FP7 Programme (compulsory) In which of the following areas did each Specific Programme of FP7 generate most impact? For each of the 6 areas per specific programme listed below, please select the 3 areas which in your view generated most impact and rank them accordingly (3 generating most impact) COOPERATION Specific Programme -Impact on scientific excellence -Impact on Technological or social innovations -Economic impact -Societal impact -Environmental impact -Regional impact -I don't know IDEAS Specific Programme

-Impact on scientific excellence -Impact on Technological or social innovations -Economic impact -Societal impact -Environmental impact -Regional impact -I don't know PEOPLE Specific Programme -Impact on scientific excellence -Impact on Technological or social innovations -Economic impact -Societal impact -Environmental impact -Regional impact -I don't know CAPACITIES Specific Programme -Impact on scientific excellence -Impact on Technological or social innovations

-Economic impact -Societal impact -Environmental impact -Regional impact -I don't know 4.2 Impacts of each Specific FP7 Programme – Comments (optional) Please specify the reasons for the ranking given under question 4.1 and/or refer to any further evidence on impact (scientific, behavioural, technological, innovation, structural, policy, and other ) FP7 has had. 4.3 Based on your experience to what extent did FP7 research activities produce enduring impact? (compulsory) -High -Medium -Low -Don't know 4.4 To what extent did FP7 research activities produce enduring impact for you as FP7 beneficiary (eg networking, bench marking, joint agenda setting, harmonisation of peer review systems? – Comments (compulsory) 4.5 Contribution of FP7 activities to the European Research Area (ERA) (compulsory)

To which of the following ERA areas did FP7 activities contribute most? Please rank the following areas on a scale from 1 – 5 (5 being the area to which FP7 activities contributed most). -More effective national research systems -Optimal transnational co-operation and competition -An open labour market for researchers -Gender Equality and gender mainstreaming in research -Optimal circulation, access to and transfer of scientific knowledge 4.6 Contribution of FP7 activities to the European Research Area (ERA) – Comments (optional) Please specify the reasons for the ranking given under question 4.5 5. European added value 5.1 EU added-value of FP7 (compulsory) In which of the following dimensions of EU added-value has FP7 been most successful? Out of the 9 EU added-value areas identified below, please select the 3 which in your view have been most successful and rank them accordingly (3 being the EU added-value dimension in which FP7 has been most successful); -Tackling pan-European challenges -Coordination of national research policies -EU scale of dissemination of research results -Pooling of resources (achieving critical mass; economies of scale and scope) -Reduction of research risk / of commercial risk -Increase competition in research

-Leverage on private investment / on public investment -Improving of S&T capabilities -Enhance researchers' mobility 5.2 EU added-value of FP7 – Comments (optional) Please specify the reasons for the ranking given under question 5.1 6. Final questions 6.1 What are the key achievements/strengths of FP7 in particular? (compulsory) 6.2 Are there shortcomings in FP7 that you think should be corrected? According to your experience have these already been addressed to in the Horizon 2020 Programme? (compulsory) 6.3 Overall participatory experience (compulsory) -Do you attend to participate again in future? yes/no -Did you know your partners in the project before? yes/no

-Did you participate for the first time? yes/no -Do you intend to stay in touch with the partners of your project after the end of the research work? yes/no How quick do you think the result of your research will lead to marketable products and services? -Within 1 year; -1 – 5 years, -5 – 10 years, -In more than 10 years 6.4 Overall are you satisfied with FP7? (compulsory) -Very satisfied -Satisfied -Moderately dissatisfied -Very dissatisfied -Don't know 3.Methods and Analytical models used in preparing the evaluation This meta evaluation builds on a significant compilation and assessment of evidence base consisting of three combined streams of information, gathered from both internal (Research and Innovation family DGs) and external sources (European RTD Evaluation Network and from the stakeholder consultation):

Statistics data were extracted from the EC databases CORDA and SESAM RESPIR CORDA, the common research data warehouse, is the Framework Programmes' (FP) central repository of data collected and/or derived during the course of FP implementation. As of 20 August 2015, CORDA information about 26,079 grant agreements. Figures are refreshed on a daily basis. DG Closed Signed Under Preparation Total* DG RTD 2686 4950 1 4964 DG CNECT 1412 2471 0 2471 DG ENER 43 133 4 137 DG MOVE 19 49 0 49 DG EAC 3 4 0 4 DG TREN 0 0 1 1 REA 6808 12613 175 12831 ERCEA 827 4567 4 4584 GSA 0 0 84 84 CS2 73 483 23 510 FCH2 4 155 0 155 HOME 26 93 0 93 ENIAC 0 63 0 63 ARTEMIS 0 38 20 58 IMI 0 56 1 57 GROW 13 16 0 16 NA 0 0 2 2 Total 11914 25691 315 26079* *including cancelled Number of FP7 grant agreements present in CORDA (20 August 2015) The SESAM Research Performance and Impact Reporting (RESPIR) tool was launched in 2012 by DG RTD. It presents, for the first time in the history of the Framework Programmes implementation, detailed statistical data on research outputs and impacts (peer-reviewed applications, applications for patents, gender, etc.) based on FP7 project final reports that are submitted and registered in the SESAM application. For FP7, the reference population in SESAM RESPIR comprises processed final reports of all DG RTD and REA FP7 projects. The tool therefore does not cover the whole FP7 as it does not include, for FP7, the activities managed by Directorate-Generals for Communication Networks, Content and Technology (CONNECT), Energy, Transport and Mobility (MOVE), the European Research Council (ERC) and some Joint Technology Initiatives (IMI, ENIAC, ARTEMIS). To illustrate this, and as noticed in the Seventh FP7 Monitoring Report (2013) referred to in this evaluation, as of 1 December 2014, more than 25,000 grant agreements were signed by the various Commission Directorates-Generals, executive agencies and other services implementing FP7. SESAM RESPIR included statistics from 7,288 project final reports (out of 8,576 closed projects). SESAM RESPIR therefore reported on 41% of FP7 projects . These projects were distributed as follows: Priority Area Number of Signed Grant Agreements Processed Final Reports Number % COOPERATION Health 1.008 400 40% Food, Agriculture and Fisheries, and Biotechnology 516 185 36% Nanosciences, Nanotechnologies, Materials and new Production Technologies 804 350 44% Energy 239 105 44% Environment (including Climate Change) 494 216 44% Transport (including Aeronautics) 589 280 48% Socio-economic sciences and Humanities 253 131 52% Space 267 111 42% Security 319 79 25% General Activities 25 11 44% Joint Technology Initiatives ( 609 121 20% Total : COOPERATION 5.123 1.989 39% Marie-Curie Actions 10.715 4.361 41% Total : PEOPLE 10.715 4.361 41% CAPACITIES Research Infrastructures 198 88 44% Research for the benefit of SMEs 1.030 471 46% Regions of Knowledge 84 46 55% Research Potential 206 107 52% Science in Society 183 90 49% Support for development of research policies 27 16 59% Activities of International Cooperation 157 60 38% Total : CAPACITIES 1.885 878 47% Fusion Energy 4 3 75% Nuclear Fission and Radiation Protection 134 57 43% Total : EURATOM 138 60 43% Total 17.861 7.288 41% Processed Final Reports in FP7 Grant Agreements by Priority Area and Funding Scheme (1/12/2014) Inputs provided by stakeholders in the 202 responses to the online public consultation Please see Annex 2. An extensive literature review This integrates: A wide corpus of thematic and horizontal evaluation studies which have been carried out over FP7 and which have been collected from the Research and Innovation family DGs (see Annex 4) These evaluations rely on a combination of qualitative and quantitative methodologies. Quantitative methodologies mostly include surveys, descriptive statistics, scientometrics; qualitative analyses methods include mostly interviews and case studies, and to a lesser extent network analysis. Impact measurement techniques include patents analysis and bibliometrics and few studies have implemented microeconomic and macroeconomic modelling methods. A striking feature is that a large majority of these evaluations is based on techniques related to observational and opinion-based investigation modes (surveys, interviews of FP participants, etc.) Most of them usually involve panels, workshops or other forms of stakeholder consultations, at two different and complementary stages: early in the process to collect data, and later on, to share and discuss their preliminary findings.

Assessments provided by the Research and Innovation family DGs services through dedicated and harmonised "building blocks templates" structured along the evaluation questions (see Annex 4); The evaluation of each of the four FP7 Specific Programmes provided by the four Supporting experts to the High-Level Group carrying out the ex-post evaluation of FP7; FP7 Annual Monitoring Reports ; DG RTD Annual Reports on Programme Evaluation Activities ; Other relevant reports such as the ex-ante assessment of FP7 and the FP7 interim evaluation and other official policy documents; National impact studies collected from the European RTD Evaluation Network. Furthermore, the methods used in this meta-evaluation were an intervention logic analysis, descriptive statistics, cluster analysis and meta-analysis techniques. The review of the aboved-mentioned evaluations has a number of limitations. The main limitation relates to their scope (these evaluations are focused on a theme or a specific instrument) and their timing (even though this is less prominent issue as most of the thematic evaluations feeding into this meta-evaluation have been carried over the last two years, once the interim evaluation of FP7 was achieved). These are well-known issues for a meta-evaluation. The main consequence is that it is not possible to aggregate their findings (using for instance an arithmetically based method to derive results at the level of the FP). This was these limitations were mitigated by strong cooperation with the services involved in the Inter service Group set up for this evaluation and by using the same template to collect data from their evaluation studies. This serves as a matrix design. Overview of the evaluation methods mostly used in the context of this meta-evaluation: - Statistical methods. - Econometric modelling. - Sociometrics (e.g. Social network analysis). - Scientometrics & Informetrics (e.g. bibliometric & patent analysis). - Judgement-based and critical methods (e.g. Surveys, interviews). 4.List of evaluation studies

List of evaluations Area/Activity SP Theme/Area Title FP7 COOPERATION Health Ex-post evaluation of the Health theme in FP7 Review of Public Health Research Projects Financed under the Commission’s Framework Programmes for Health Research Report of the Independent Expert Group on the Future of European Public Health Research Impact Assessment of the Health Theme and proposal for post 2013 EU support for health research and innovation Profiles, motivations and expectations of participants to EC funded research in Health (2002–2010): A statistical analysis Evaluation of Health research under FP6 and FP7, In-depth Case studies Energy Mid-term evaluation on the impact of energy EU-funded research and demonstration projects under FP6 and FP7 Evaluation of the impact of projects funded under the 6th and 7th EU

Framework Programme for RD&D in the area of non-nuclear energy Transport Interim Evaluation of EU FP7 Transport Research, notably within Theme 7 of the Cooperation Programme "Transport (including Aeronautics)" TRI VALUE - Ex-post evaluation of the Transport (including Aeronautics) theme of the FP7 Cooperation Specific Programme Environment Research on climate change funded by the Seventh Framework Programme Stock-Taking of Results and Impacts of EU-Funded Environmental Research State of the Art and Forward-Looking Analysis of Environmental Research and Innovation KBBE Ex-post evaluation and impact assessment of FP7-funded projects in Food, Agriculture, Fisheries and Biotechnology Impacts of EU Framework Programmes (2000-2010) and Prospects for Research and Innovation In Food, Agriculture, Fisheries and Biotechnologies Small and Medium-sized Enterprises - Participation in FP7 Projects in the Biotechnologies Activity Meta-analysis of "Bio-Technology", "Agriculture", "Food", "Marine and Maritime" and Horizontal themes Impact Assessment of Food related EU Research in PF6 and FP7 Ex-post evaluation The Ocean of Tomorrow calls (2010-2013) NMP Ex-post evaluation and impact assessment of funding in the NMP thematic area - FP7 Cooperation Specific Programme Comparative scoreboard and performance indicators in NMP research activities between EU and third countries Analysis of patenting activity of FP7 NMP projects Strategy Definition and Road Mapping for Industrial Technologies to Address Grand Challenges Innovation - How to Convert Research into Commercial Success Story? Part 1 : Analysis of EU-funded Research Projects in the Field of Industrial Technologies Part 2 : Analysis of Innovation Successes in the Field of Industrial Technologies Ex post Evaluation of FP6-NMP - Project Level Analysis of Smart Specialisation Strategies in NMP related areas ICT Interim evaluation of the ICT research in the 7th FP Interim Evaluation of AAL Joint Programme Final Evaluation of the Ambient Assisted Living Joint Programme (AAL JP) Assessing impact of the European Innovation Partnership on Active and Healthy Ageing Ex-post evaluation of the ICT research in the Seventh Framework Programme, including the support study SMART 2013/0049 (COOP. and CAP. SP) SSH Evaluation, monitoring and comparison of the impacts of EU funded SSH (Socio-economic sciences and Humanities) research in Europe (IMPACT-EV project) Space Evaluation of socio-economic impacts from space activities in the EU Ex-post evaluation of the FP7 Space research actions Security Review of security measures in the 7th Research Framework Programme FP7 2007-2013 SMEs and their participation in security research Final report from the European Security Research and Innovation Forum (ESRIF) Ex-post evaluation of the FP7 Security research actions CAPACITIES SMEs Progress Reports on SMEs Participation in the 7th R&D Framework Programme Performance of SMEs within FP7 - An Interim Evaluation of FP7 Components Research infrastructures Assessing the Projects on the ESFRI (European Strategy Forum for Research Infrastructures) Roadmap Evaluation of pertinence and impact of the EU research infrastructure activity in FP7 Regions of Knowledge Evaluation of research intensive clusters as potential vehicles for smart specialisation in the European regions Final Evaluation of the Regions of Knowledge (RoK) Programme Research potential of Convergence Regions Analysis and recommendations for the future evolution of the Research Potential Activity of the EU-EU FP7 Capacities Programme - Expert Group on Research Potential Project Portfolio - 17 May 2011 Final evaluation of FP7 “Research Potential” (REGPOT) Programme, COWI report – April 2014 Science in Society Ex-post evaluation of Science in Society in FP7 Interim evaluation and assessment of future options for Science in Society Actions Coordination of Research Activities Ex-post evaluation of the research and development programme undertaken by several Member States aiming at supporting research and development performing small and medium-sized enterprises (Eurostars) Review of the Joint Programming Process Interim Evaluation of the European Metrology Research Programme (EMRP) Support to the coherent development of research policies Evaluation of the "Coherent Development of Research Policies" Programme ERA initiatives Expert group on open innovation and knowledge transfer Support for Continued Data Collection and Analysis Concerning Mobility Patterns and Career Paths of Researchers – MORE 2 Knowledge Transfer Study 2010-2012 Evaluation of the EURAXESS Project (2008-2012) Monitoring of the human resources policies and practices in research ("European Partnership for Researchers monitoring" EPR ) Study to monitor the implementation and impact of IP recommendation and code of conduct - Knowledge transfer Analysis of the ERA State of play in Member States and Associated Countries: focus on priority areas Evaluation of ERA progress in MS and AC Researchers' report 2014 International cooperation International Science and Technology Cooperation in the EU's Seventh Framework Programme: the specific programme 'Cooperation' and its thematic areas European Added Value of EU Science, Technology and Innovation actions and EU-Member State Partnership in international cooperation Mapping of best practice regional and multi-country cooperative STI initiatives between Africa and Europe — identification of financial mechanism(s) 2008–2012 Overview of International Science, Technology and Innovation Cooperation between Member States and Countries outside the EU and the Development of a Future Monitoring Mechanism International Cooperation in Science, Technology and Innovation: Strategies for a Changing World - Report of the Expert Group Established to Support the Further Development of an EU International STI Cooperation Strategy Review of S&T cooperation agreement between the European Union and Russia Review of S&T cooperation between the European Union and the USA Review of S&T cooperation between the European Union and New Zealand Review of S&T cooperation agreement between the European Union and South Africa Review of S&T cooperation between the European Union and Mexico Review of S&T cooperation between the European Union and Morocco Review of S&T cooperation between the European Union and the Federative Republic of Brazil Review of S&T Cooperation Agreement between the European Union and Government of the Republic of India Review of the S&T Agreement between the European Union and the Republic of Korea Review of S&T cooperation between the European Union and Tunisia Review of S&T cooperation between the European Union and Argentina Review of S&T cooperation between the European Union and Chile PEOPLE Contribution to the interim evaluation of Marie Curie 2014-2020 + ex-post of Marie Curie under FP7 FP7 Marie Curie Life-long Training and Career Development Evaluation: Individual Fellowships and Co-funding Mechanism Marie Curie researchers and their long-term career development: A comparative study FP7 Marie Curie Actions Interim Evaluation Ex-post Impact Assessment study concerning the ‘Marie Curie Actions’ under the Sixth Framework Programme IDEAS Comparative overview of the current research funding instruments in Europe and selected countries Monitoring ERC's implementation of excellence Understanding and assessing the impact and outcomes of the ERC funding schemes Emerging Research Areas and their Coverage by ERC-supported Projects Comparative scientometric assessment of the results of ERC funded projects EURATOM Ex-post evaluation of the Euratom Framework Programme (2007-2013) JRC Ex-post evaluation of FP7 JRC activities PPPs Interim Assessment of the Future Internet Public-Private Partnership (2nd) Interim Assessment of the Future Internet Public-Private Partnership Interim Assessment of the Research Public Private Partnerships in the European Economic Recovery Plan: Energy-Efficient Buildings, Factories of the Future, and European Green Cars Initiative Final Assessment of the Research PPPs in the European Economic Recovery Plan: Factories of the Future; Energy-efficient Buildings; European Green Cars Initiative JTIs Interim Evaluation of the Innovative Medicines Initiative Joint Undertaking (IMI JU) First Interim Evaluation of JTI ENIAC and ARTEMIS First Interim Evaluation of the Fuel Cells and Hydrogen Joint Undertaking (FCH JU) First Interim Evaluation of the Clean Sky Joint Undertaking (CS JU) Commission Response to the First Interim Evaluation of CS, FCH, IMI Joint Untertakings Commission Response to the First Interim Evaluation of ARTEMIS and ENIAC Joint Technology Initaitives Second Interim Evaluation of the Innovative Medicines Initiative Joint Undertaking (IMI JU) Second Interim Evaluation of the Fuel Cells and Hydrogen Joint Undertaking (FCH JU) Second Interim Evaluation of the Clean Sky Joint Undertaking (CS JU) Commission Response to the Second Interim Evaluation of CS, FCH, IMI Joint Untertakings ARTEMIS and ENIAC Joint Technology Initatives Second interim evaluation Commission Response to theSecond Interim Evaluation of ARTEMIS and ENIAC Joint Technology Initiatives EIP Outriders for European Competitiveness: European Innovation Partnerships (EIPs) as a Tool for Systemic Change RSFF First Interim Evaluation of the Risk-Sharing Financial Facility (RSFF) Second Interim Evaluation of the Risk-Sharing Financial Facility (RSFF) Executive agencies External Evaluation of the Research Executive Agency (REA) and the European Research Council Executive Agency (ERCEA) Horizontal studies Innovation Union Interim Evaluation of the Innovation Union Impact Assessing the contribution of the Framework Programmes to major innovations Assessing the contribution of the Framework Programmes to the development of human research capacity Assessing the research management performance of FP projects Understanding the long term impact of the Framework Programmes Participation Network analysis of FP participation The role and participation of industry in the FPs The role and participation of research organisations in the FPs The role and participation of universities in the FPs Interim evaluation of the participation of SMEs in the Cooperation Programme and the Research for the benefit of SMEs schemes under the Capacities Programme of FP7 (2007-2013) Network analysis of Civil Society Organisations' participation in research framework programmes Gender SHE Figures 2012 SHE Figures 2015 Simplification Assessing the Effectiveness of Simplification Measures under FP7 Other activity reports Monitoring Reports Annual Monitoring Reports of FP7 Innovation Union Competitiveness reports Programme Evaluation Activities DG RTD Annual Report on Programme Evaluation Activities 2011 DG RTD Annual Report on Programme Evaluation Activities 2012 DG RTD Annual Report on Programme Evaluation Activities 2013 National impact studies are listed in Annex 21. 5. Evaluation questions Evidence Building Block 1: Rationale (“why”): Building Block 1 aims to describe the logic of intervention (of the expenditure programme); to analyse the relevance of its objectives; whether the objectives are consistent with the strategic context and the identified challenges in the FP7 period 2007-2013. Evidence Building Block 2: Implementation (“how”): Building Block 2 aims to describe the FP7 intervention from a process/implementation perspective (how was the programme in your thematic area implemented?) by factually establishing the participation patterns by research actor type; the distribution of funds by research actor type; the changes over different FPs (FP7, FP6, FP5, and prior if relevant); and by analysing the reasons for these patterns, distributions and trends. Evidence Building Block 3: Achievements (“what”) – direct achievements (“outputs”): Building Block 3 describes the “outputs” or direct achievements of “all what is produced” through this intervention at project and programme level per thematic research area or main topic. Building Block 3 also describes and analyses the uptake on cross-cutting issues per thematic research area. Evidence Building Block 4: Achievements (“what”) – wider achievements (“results” and “impacts”): Building Block 4 deals with the “results” of the intervention for the beneficiaries and the wider economic, societal and environmental “impacts” for Europe at large. Evidence Building Block 5: European Added Value: Building Block 5 deals with European Added Value (EAV). EAV refers to the need for Europe to intervene. EAV is analysed through establishing a need for public intervention (as opposed to market forces) and establishing a need for this public intervention at EU level (as opposed to MS and regional level). Evidence Building Block 6: Conclusions on FP7 and Outlook for H2020: Building Block 6 wraps up the analysis and concludes per thematic research area the main findings for FP7 and indicates what can be learned from FP7 achievements for the successor programme H2020 and what FP7 shortcomings are already addressed in H2020. Examples of the specific questions used to build the evidence base under the “building blocks’ include the following. Guiding questions: Has, for each specific programme, clear objectives been formulated? Were the programme objectives adequately designed to address EU needs and societal challenges as intended? Were the activities and budgets allocated among the areas sufficiently and adequately distributed, in a way that strengthened the scientific and technological base and encouraged international competitiveness? How have the objectives and coverage of topics evolved through time, and how did they align to the overall EU policy context? Has the programme evolved to become less or more prescriptive regarding research and innovation topics and to allow bottom up approaches to deliver innovative ideas? Did the priority-setting evolve in this respect during FP7?

Have project outputs lead to research training and capability effects? innovation (e.g. innovative new commercial products, profitable new services or starts-ups)? Which fields and which clients/segments of society have benefitted most in this regard? What were the reasons for success, what are the areas where there is scope for improvement? Has the programme sufficiently promoted the translation of research and innovation into market applications? Has the programme supported the demonstration of the market potential of new products or processes? Has the programme supported policy - making, including standardisation and legislation? Has the programme had a positive leverage effect in promoting national research efforts? Has it contributed to a better exploitation of results? Which socio-economic, environmental and other relevant impacts resulted from FP7-funded research projects? What is the causal link between them? To what extent did FP7 Specific Programmes help address the main societal, environmental and economic challenges? To what extent has it led to innovations that contribute to improving quality of life? Did FP7 Specific Programmes enable all research and innovation stakeholders – research institutions, researchers, citizens, policy makers, industry, and third sector organisations – to make a full contribution towards the realisation of its objectives? Overall, did FP7 Specific Programmes help meet the needs, expectations and values of European citizens? How have the FP7 Specific Programmes influenced research and related policies? To what extent would researchers have been able to undertake the research project funded through FP7 in the absence of this EU-level funding? To what extent has FP7 yielded outputs, results and impacts that intervention at Member State level alone could not have achieved? How did EU support for research add value compared to purely national public funding in terms of elements such as the following? Pooling of resources and creating EU-Networks(achieving critical mass; economies of scale and scope) Leverage on public and private investment Reduction of research and commercial risks Improving of S&T capabilities and pan-European cooperation

Accelerating the generation of knowledge and research outputs Increase competition in research Tackling pan-European challenges Coordination of national research policies. 6.Background to FP7 The Seventh Framework Programme was designed in a context of the reinvigorated Lisbon Strategy process which had set the European Union a strategy goal to become the most competitive and dynamic knowledge-based economy of the world, capable of sustainable economic growth with more and better jobs and greater social cohesion . The overriding objective of FP7 was to contribute to the Union becoming the world's leading research area. FP7 was also intended to support progress towards the 3% GDP target in R&D by 2010, two-thirds of which should come from the private sector , as established by the Barcelona European Council in March 2002. The specific objectives of FP7 were to support the creation of the European Research Area (ERA) and contribute to the development of a knowledge-based economy and society of Europe. While building on the Sixth Framework Programme, FP7 was thus attributed more ambitious objectives more closely linked with the economic and societal challenges the EU was facing (decelerating economic growth, fiercer international competition supported by rapid advances of new technologies such as ICT, significant environmental degradation caused by global warming and climate change). These new objectives were also established to address three main challenges Europe was facing in the area of R&D: a low level of investment in R&D (1.97% of GDP) notably as compared with the USA (2.59%), a “brain drain” effect leading the best researchers to move abroad, and a deficient capacity in transforming basic research results into marketable innovations, making an economic success of them: "European companies apply for 170 patents each year per million inhabitants compared with 400 for American companies. And the Union's commercial deficit for high-tech products is approximately €23 billion per year ". The following table gives an overview of the expected impacts of the "new FP7" compared with the "do nothing" option. Expected impacts of the new FP7 and the do nothing option (business as usual scenario taken as a reference)



IMPACTS POLICY OPTIONS "do nothing" "new FP7" Expected aggregate impacts on the achievement of the Lisbon, Barcelona and other community objectives LISBON OBJECTIVES Economic growth performance In the long run, up to 0.84 percent of GDP lost compared to the business- as-usual scenario In the long run, between 0.45 and 0.96 percent extra GDP is generated compared to the business-as-usual scenario, because of crowding-in and rates of return/multiplier effects. The literature shows that the crowding-in effect of €1 of public R&D funding allocated to business has been estimated to range between €0.7 and € 0.93. The private rates of return to private R&D can be as high as 43 percent, the social ones as high as 160 percent. The rates of return to publicly-funded research could be as high as 67 percent. Employment creation In the long run, up to 800,000 jobs lost compared to the business-as-usual scenario In the long run, between 400,000 and 925,000 extra jobs are created compared to the business-as-usual scenario. The literature shows that the rate of growth of total factor productivity (TFP - due to improvements in the efficiency of production or to pure technological progress) has a positive impact on the employment rate, with a one-year lag, and that both in the short- and long-term, countries with higher than average TFP growth tend also to have higher than average growth in employment. Competitiveness In the long run, extra-European exports lower by up to 2 percent, imports higher by up to 1.43 percent compared to the business-as-usual scenario In the long run, extra-European exports could be higher by between 0.64 and 1.57 percent; imports lower by between 0.3 and 0.9 percent compared to the business-as- usual scenario. The literature shows that publicly funded research is critical for the development of new products, processes and services. Increases in R&D also increase productivity. BARCELONA OBJECTIVES R&D intensity In the long run, Europe's R&D intensity lower by up to 0.1 percent of GDP compared to the business-as-usual scenario In the long run, Europe's R&D intensity higher by between 0.059 and 0.23 percent of GDP compared to the business-as-usual scenario. This is because of high crowding-in effects (see above under economic growth performance) Research employment In the long run, up to 87,000 jobs lost compared to the business-as-usual scenario In the long run, between 40,000 and 215,000 extra jobs compared to the business-as- usual scenario. OTHER COMMUNITY POLICIES Göteborg strategy Less informed design of EU Sustainable Development Strategy and disorganised consideration of the three pillars of sustainability Knowledge-based design of the EU Sustainable Development Strategy and more balanced consideration of the three pillars of sustainability in the decision-making process; EU evidence-based leadership in international negotiations Other Community Policies More ad hoc and inefficient development of perhaps less effective Community policies Easier development of more evidence-based and effective policies in the fields of agriculture, economic and financial affairs, employment, enterprises, environment, fisheries, food, health, maritime affairs, etc. Expected impacts of Specific Programmes SPECIFIC PROGRAMME: PEOPLE People Less European mobility and cooperation; less attractive scientific careers for European citizens (in particular women); Europe less attractive to the best foreign researchers; reduced level and diversity of skills of individual researchers; less sustainable linkages between academia and industry, and across disciplines More research can be carried out in Europe; research will generally be of higher quality, more inter-disciplinary, and where appropriate take industry better into account SPECIFIC PROGRAMME: IDEAS Ideas More national, non-competitive and overlapping funding; fewer scientific publications; publications of lower quality and fewer citations as the competition for basic research funding between individual research teams remains organised at national level, i.e. essentially meaningless in highly specialised fields of science in most countries A better and enlarged knowledge base for European enterprises on which the innovation of products and process can be based; levelling-up effects as incentives are provided to increase institutional and researcher capabilities, produce better research proposals, and carry out higher-level research; structuring effects (dissemination; increased attractiveness ERA) SPECIFIC PROGRAMME: COOPERATION Expected impacts of Specific Programmes Collaborative research Greater fragmentation and inefficiency of research efforts in Europe; fewer projects carried out by research teams on a European scale and limited to resources and knowledge available at national level; more slow advance in important fields of science; in some countries, capabilities in particular research fields declining due to inadequate interaction with top teams located elsewhere Some research activities are of such a scale that no single Member State can provide the necessary resources and expertise. In these cases, EU projects can allow research to achieve the required "critical mass", while lowering commercial risk and producing a leverage effect on private investment. EU-scale actions also play an important role in transferring skills and knowledge across frontiers. This helps to foster excellence in research and development through enhancing capability, quality and EU-wide competition, as well as improving human capacity in S&T through training, mobility and European career development. EU support can also contribute to a better integration of European R&D, by encouraging the coordination of national policies, by the EU-wide dissemination of results, and by funding research for pan-European policy challenges. JTIs Reduced competitiveness of European industries; reduced participation of industry in the FP; negative signal given to knowledge-intensive and hightech industries Important contribution made to the achievement of the Lisbon and Barcelona agenda through the formulation for areas critical for European competitiveness of ambitious, long-term and strategic research and wider policy agenda, the commitment of a critical mass of financial, organisational and human resources under public-private partnerships, indicatively sharing costs in a 1/3-2/3 format. International cooperation Europe reneges on its commitments in international fora and goes entirely against the trend whereby other industrialised countries/regions are seeking to expand their international S&T cooperation. Socio-economic development and global competitiveness stimulated; contributions made to Europe's many key international commitments (e.g. Kyoto, Convention on Biological Diversity, Biosafety Protocol, the plan of Implementation adopted at the World Summit on Sustainable Development). Coordination of national research programmes Return to the complete fragmentation of the pre-ERA period, with 25 MS and numerous regions defining their research priorities independently from each other and from the EU; waste of already scarce resources; opportunity lost to restructure the European research fabric so as to enhance EU competitiveness Strong contribution made to the restructuring of the European research fabric in a coordinated and organised way and to the development of ERA. Expected impacts of Specific Programmes SPECIFIC PROGRAMME: CAPACITIES Research infrastructures; Research for the benefit of SMEs; Regions of knowledge; Research potential; Science in society; Specific activities of international cooperation • Realising full potential Increased inefficiency and fragmentation of the European research landscape; less coordination of efforts, less possibility to share costs and access, potential duplication, loss of research capability European SMEs deprived of important resources and opportunities to remain competitive in a global economy Better efficiency of public funds and stimulation of increased synergies between public and private funds; seamless access to all kinds of resources spread throughout Europe and the world. The exploitation by SMEs of their research improved, EU-wide transfer of technology; research results potentially transformed into products and services Expected impacts of management choices Administrative burden No administrative burden Limited administrative burden; cost of participation reduced; procedures simplified and rationalised The R&D policy context evolved during the implementation of FP7 with the adoption of new initiatives to stimulate the European Research Area on the one hand, and of the Europe 2020 Strategy and Europe Economic Recovery Plan on the other hand. A first batch of initiatives to stimulate the ERA was launched in 2007-2008 as a follow-up of the ERA Green Paper published on 4 April 2007 : Recommendation on the management of intellectual property in knowledge transfer activities and Code of Practice for universities and other public research organisations . This recommendation was endorsed by a resolution adopted by the Competitiveness Council on 30 May 2008 . In June 2008 launch of EURAXESS branding to adapt to new initiatives and ERA needs and to duly respond to users' needs. Communication 'Better careers and more mobility: a European partnership for researchers'. Related Council conclusions were adopted on 26 September 2008 .

In November 2008 the Commission launched the Human Resources Strategy for Researchers implementing the Commission Recommendation 2005 of the European Charter for Researchers and the Code of Conduct for their Recruitment. Communication 'Towards Joint Programming in Research: Working together to tackle common challenges more effectively' . The Competitiveness Council adopted conclusions on this issue on 2 December 2008 . Proposal for a Council regulation on the Community legal framework for a European Research Infrastructure (ERI) . This regulation was adopted by the Council on 25 June 2009 . Communication 'A Strategic European Framework for international science and technology Cooperation' . It was endorsed by Council conclusions adopted on 2 December 2008 . Building on these elements and aiming at meeting the demand of the European Council (ERA to be completed by the end of 2014), the Commission adopted on 17.7.2012 the Communication 'A Reinforced European Research Area Partnership for Excellence and Growth' . Related Council conclusions were adopted on 12.12.2012 . A first Progress Report was published before the end of FP7 . Most of the actions by the Commission that were announced in the Communication have been completed such as the establishment of open access to scientific publications as a general principle for Horizon 2020, a Communication on access to and preservation of scientific information in the digital age, the launch of RESAVER or are still ongoing as they are part of a continuous process like open access to scientific data, e-science, open innovation, Joint Programming, ESFRI etc. Furthermore, as a response to the economic crisis, the following initiatives were adopted that had an impact on the objectives and activities of FP7: As part of the European Economic Recovery Plan , three contractual Public-Private Partnerships (Factories of the Future, Energy-efficient Buildings and Green Cars) were established in 2009 to develop new technologies for sectors which have experience significant downturns in demand a s a result of the economic crisis and to foster the transition to a sustainable economy. The Europe 2020 Strategy for smart, sustainable and inclusive growth was adopted in March 2010 to foster growth and competitiveness by developing an economy based on knowledge and innovation, which would be more resource efficient and greener, and achieve high employment, social and territorial cohesion. Amongst the seven Europe 2020 flagship initiatives, the Innovation Union was launched in October 2010 . Simplification measures were introduced to facilitate the award and management of grants in order to realise European research potential both in Europe and elsewhere. A detailed intervention logic is presented hereafter. Intervention Logic of FP7





Source: DG RTD FP7 was the cornerstone of EU research policy and a major component of the EU innovation system which includes the following other programmes , smaller by their budgets and scopes, which ran in parallel, with their specific objectives, creating a complex governance structure: The Competitiveness and Innovation Framework Programme (CIP) included three programmes: Enterprise and Innovation, Intelligent Energy Europe, and ICT policy (EU budget contribution of €3.6 billion for 2007-2013). The European Institute for Innovation and Technology (EIT) brought together higher education, research and business to stimulate innovation in Knowledge and Innovation Communities (EU budget contribution of €309 million for 2007-2013). The Structural Funds (ERDF, ESF, Cohesion Fund) allocated to R&D and innovation, entrepreneurship, ICT and human capital development (EU budget contribution of €86 billion for 2007-2013) Lifelong Learning Programme (LLP) included Comenius, Leonardo da Vinci, Erasmus, Erasmus Mundus, Grundtvig, E twinning, Transversal Actions (EU budget contribution of € 6.2 billion for 2007-2013). The range of funding schemes implemented in FP7 was wide, including six schemes:

- Collaborative projects were focused research projects with clearly defined scientific and technological objectives and specific expected results (such as developing new knowledge or technology to improve European competitiveness). They were carried out by consortia made up of participants from different countries, and from industry and academia. - The Networks of Excellence were designed for research institutions willing to combine and functionally integrate a substantial part of their activities and capacities in a given field, in order to create a European "virtual research centre" in this field. - Coordination and Support actions covered not the research itself, but the coordination and networking of projects, programmes and policies. This included, for example coordination and networking activities, dissemination and use of knowledge, support for transnational access to major research infrastructures, actions to stimulate the participation of SMEs, civil society and their networks, support for cooperation with other European research schemes (e.g. "frontier research"). - Individual projects were carried out by individual national or multinational research teams, led by a "principal investigator", funded by the European Research Council (ERC). - Support for training and career development of researchers from across the European Union and its research partners were implemented in the Marie Curie action. - Research for the benefit of specific groups – in particular SMEs targeted research and technological development projects where the bulk of the research was carried out by actors such as universities, research centers or other legal entities, for the benefit of specific groups, in particular SMEs, or for civil society organisations and their networks. In FP7, the cost calculation regime saw a number of changes as compared with FP7, as illustrated in the tables below. In particular: each beneficiary has a specific Indirect Cost Model (ICM), and the use of lump sums, flat rates and average personnel costs were gradually implemented. Furthemore, a participants guarantee fund was introduced, the number of ex-ante financial capacity checks was reduced (in FP6, all non-public beneficiaries had to be checked, in FP7 only the ones requesting EU contributions > € 500,000), and the number of certificates on financial statements to be provided decreased (in FP6 each cost claim had to be accompanied by a certificate of an auditor, in FP7 a threshold of € 375,000 was introduced). RTD Schemes CSA Schemes Real Indirect Costs (if analytical accounting system) Real Indirect Costs Real Indirect costs (to be declared with justification and method calculation) Simplified Method (if accounting system allows to identify all indirect costs) Real Indirect Costs Real Indirect costs (to be declared with justification and method calculation) Provisional flat rate 60% 20% Standard flat rate 20% 20% (to be declared – single exception to actual/real costs declaration) Maximum reimbursement Indirect Costs CSA Scheme 7% (applied by EC during calculation of payment) Indirect Costs in FP7 Source: DG RTD



Maximum reimbursement rates Research and Technological Development Activities (*) Demonstration Activities Other Activities Network of excellence 50% 75% (**) 100% Collaborative project (****) 50% 75% (**) 50% 100% Coordination and support action 100% (***)

Reimbursement Rates in FP7 Source: DG RTD (*) Research and Technological Development includes scientific coordination. (**) For beneficiaries that are non-profit public bodies, secondary and higher education establishments, research organisations and SMEs (***) The reimbursement of indirect eligible costs, in the case of coordination and support actions, may reach a maximum 7% of the direct eligible costs, excluding the direct eligible costs for subcontracting and the costs of resources made available by third parties which are not used on the premises of the beneficiary. (****) Including research for the benefit of specific groups (in particular SMEs).



7.Overview of FP7 indicators Indicator Target Results/latest state of play 1 Projects that achieved all or most of their objectives… 90% (by 2013) 98 % by Sep 2015 Source: CORDA/SESAM 2 ... of which projects that achieved all of their objectives 75% (by 2013) 47 % by Sep 2015 Source: CORDA/SESAM 3 Share of EU financial contribution to Industry (*) The target depends on the specific thematic area under the Specific Programme "Cooperation" (between 40% for NMP and 3 % for SSH) (by 2013) 24,6 % by Oct 2014 Source: CORDA 4 Share of EU financial contribution to SMEs 15% (by 2013)1 17.4 % by Oct 20132 Source : AAR 2013 5 Projects producing specific outputs disseminated to policy makers 75% (by 2013) 95% by Nov 2013 Source : AAR 2013 6 Number of international prizes and awards to ERC grant holders 200 (by 2020) 134 by July 2013 Source : AAR 2013 7 Number of scientific publications by ERC grant holders ~40,000-60,000 (by 2020) ~20,000 by Dec 2013 Source : AAR 2013 8 Number of international scientific users having benefited from access to Research Infrastructures 30,000 (by 2013) 18,300 by Nov 2013 Source : AAR 2013 9 Percentage of users satisfied with services offered by research infrastructures participating in Integrating Activities (good to very good overall appreciation) >97% (by 2013) 97% by Nov 2013 Source : AAR 2013 1 = For the budget of the Cooperation SP, the following activities are not included: grants to the European Space Agency (ESA), JTIs, General Activities such as the CORDIS services, the horizontal ERA-NET scheme, research organisations in the EU, strengthened coordination with EUREKA, scientific and technological cooperation activities carried out in the COST and the European Metrology Research Programme.



8.Background on current situation Since the launch of FP7, the economic context has changed dramatically. A recession triggered by the 2008 financial crisis led to the adoption of stimulus packages to kick-start the economy. The key challenge is to stabilise the financial and economic system in the short term while also taking measures to create the economic opportunities of tomorrow. The successor of FP7, Horizon 2020, was launched in the beginning of 2014. It brings together all existing Union research and innovation funding, including the framework programme for research, the innovation related activities of the competitiveness and innovation framework programme and the European Institute of Innovation and Technology (EIT). Horizon 2020 has a number of new features that make it fit for purpose to promote growth and tackle societal challenges. These include: - Major simplification through a simpler programme architecture, a single set of rules, less red tape through an easy to use cost reimbursement model, a single point of access for participants, less paperwork in preparing proposals, fewer controls and audits, with the overall aim to reduce the average time to grant by 100 days;

- An inclusive approach open to new participants, including those with ideas outside of the mainstream, ensuring that excellent researchers and innovators from across Europe and beyond can and do participate; - The integration of research and innovation by providing seamless and coherent funding from idea to market; - More support for innovation and activities close to the market, leading to a direct economic stimulus; - A strong focus on creating business opportunities out of our response to the major concerns common to people in Europe and beyond, i.e. ‘societal challenges’; - More possibilities for new entrants and young, promising scientists to put forward their ideas and obtain funding. Horizon 2020 focuses resources on three distinct, yet mutually reinforcing, priorities, where there is clear Union added value: i) Excellent Science, ii) Industrial Leadership and iii) Societal Challenges. 9. The Cooperation Specific Programme Objectives The Cooperation Programme was the largest single component of FP7, and invested just over € 32 500 million (65% of the total available budget) across a 7-year term, through a combination of collaborative research and various coordination actions across 10 thematic areas: Health; Food, Agriculture and Fisheries, and Biotechnology; Information and Communication Technologies; Nano-sciences, Nano-technologies; Energy; Environment (including Climate Change); Transport (including Aeronautics); Socio-economic Sciences and Humanities; Space;

Security. The overarching aim of the Cooperation Specific Programme was to contribute to sustainable development. The overall objective was to help Europe gain or consolidate international leadership in a wide range of key scientific and technology areas, in order to ensure European competitiveness at the global level. The main objectives of the Cooperation Programme were to support: –collaborative research (through collaborative projects, networks of excellence and coordination and support actions); –public-private partnerships in research (through Joint Technology Initiatives set up as Joint Undertakings under Article 187 TFEU as well as contractual public-private partnerships directly under the Framework Programme); –coordination of non-Community research programmes (through ERA-NET and ERA-NET Plus actions as well as joint programmes with Member States under Article 185 TFEU ); and international cooperation. Effectiveness Collaborative research In the modern global economy, it can no longer be expected that single teams or even Member State can provide the necessary scale and scope of resources required to conduct research. For example, the average research expenditure of an EU-25 country is just over € 7 billion per year. (% of GDP) FP-induced collaborative research encourages trans-national partnerships, brings together resources, disciplines, scientific excellence, thus achieving a critical mass which could not be attained at national level. Participation of different actors - from university, industry and public research laboratories – and the interaction between these actors is also a key aim of EU collaborative RTD actions. The Cooperation Specific Programme was effective in attracting leading public research and private actors, with approximately 75% of participants from the public sector and 25% being from private entities. Collaborative research projects enable those research teams wishing to develop their S&T capabilities in specific fields to participate in top transnational teams, benefit from learning and synergies. In this way, FP7 had an important structuring effect on the European research system. The Programme was also effective in increasing the breadth of networks and the engagement of players from EU13 countries. In this regard, a striking aspect of FP7 networks was the increase in participant numbers compared to FP6. There were 450,000 new collaborations recorded under FP7. Moreover, almost three-quarters of organisations participating in FP7 were new to the networks, not having participated under FP6. Moreover, cross-disciplinarity of research is growing, and no Member State can be expert in all fields, especially the emerging ones. Hence researchers must increasingly look beyond their own frontiers if they want to find high-quality expertise in complementary disciplines. JTIs and contractual Public-Private Partnerships The establishment of the five Joint Technology Initiatives under Article 187 TFEU for Innovative Medicines, Clean Sky, ARTEMIS, ENIAC and Fuel Cells and Hydrogen and the contractual Public-Private Partnerships set up under the European Economic Recovery Plan in the areas of the Green Car, Energy Efficient Buildings and Factories of the Future involved the commitment of massive financial, organisational and human resources through public-private partnerships. The JTIs and contractual Public-Private Partnerships contributed to increasing industry participation and implementing industry-driven research agendas. They fostered innovation, by bringing together all the stakeholders along the innovation chain, and supporting market uptake. See Annex 15 and 17 Coordination of non-Community research programmes

The importance of the coordination of the national programmes is obvious when one considers the amount of funding concerned. The Framework Programme accounts for only 6% of the total public R&D expenditure in the EU, while, for example, the annual budget of DFG in Germany is over € 1 000 million and that of CNRS in France is over € 2 000 million. However, prior to the adoption of FP7, publicly financed research and innovation programmes remain largely uncoordinated and are still defined separately in each Member State in many regions. Before the crisis, EU funding represented more than 20 % of project based funding in Europe, and this has increased since then thanks to higher annual budgets in the Seventh Framework Programme for Research and Technological Development (FP7) . But the bulk of research remains funded at national level. This shows the necessity to better coordinate research in Europe. A significant effort has been accomplished to coordinate MS activities by developing common strategic research agendas, aligning national plans, defining and implementing joint calls. The new “ERA-NET PLUS” contributed more strongly to the restructuring of the European research fabric in a coordinated and organised way and thus to the development of the ERA. As for Article 169, the number of joint programmes increased and other fields of research were included, thus strengthening the impact on the ERA. ERA-NETs and Joint Programming Initiative (JPI) received positive feedback from national policy makers as regards the value of coordinating national research activities. The KBBE theme noted clear indications of important ‘critical mass’ formed as a result of collaboration. Success of these schemes can be demonstrated for example in the Health theme: The neurodegenerative disease research (JPND), with a €2 million EU contribution, leveraged €75 million, the AMR JPI leveraged €13.8 million via a €1.9 million EU contribution. Six Era-NETs leveraged €119 million via an EU contribution of €15 million. Overall, in ERA-NETs and ERA-NET Plus, the five most active participating countries accounted for 40% of the participations (FR, DE, NL, AT, DK). See Annex 16 and examples in the following Annexes. International cooperation Through the Cooperation programme, contributions was made to Europe's many key international commitments, in particular FP7-Environment played a key role in the development and aggregation of climate change models, with a strong contribution to the International Panel on Climate Change (IPCC). While models could be developed at national level, FP7-Environment provided a unique coordination role, allowing models to be run together, ensuring the completeness of