THE COURT’S OBSERVATIONS

INTRODUCTION

3.1. The subject of this chapter is performance (see Figure 3.1 ). It continues the theme — which has been a feature of our annual reports since 2010 — of including a chapter assessing different aspects of the systems put in place by the Commission to ensure that EU funds are spent well, and not just in line with relevant legislation.

3.2. Since 2010, we have analysed the Commission’s self-assessment on performance as stated in the annual activity reports (AARs) of a selection of the Commission’s directors-general, and included an examination of some of the themes developed in that year’s special reports on performance audit. Other elements of performance that we have examined are, the management plans of selected directorates-general (DGs) and the reporting by the Commission to the European Parliament and the Council including the first few editions of the Commission’s ‘evaluation reports’ (1).

Figure 3.1 – Performance in the EU Performance in the EU is assessed on the basis of the sound financial management principles (economy, efficiency and effectiveness) (2), and covers: (a) inputs — financial, human, material, organisational or regulatory means needed for the implementation of the programme; (b) outputs — the deliverables of the programme; (c) results — the immediate effects of the programme on direct addressees or recipients; (d) impacts — long-term changes in society that are, at least partly, attributable to the EU’s action.

3.3. In the equivalent chapter of last year’s annual report, we took a particular look at the EU’s long-term strategy, Europe 2020. We examined the EU’s high-level monitoring and reporting of the strategy and, for the European Structural and Investment Funds (3) (the ESI funds), how the strategy feeds through into the partnership agreements and funding programmes agreed between the Commission and Member States. This year we continue that theme, and focus our examination of the Commission’s performance through Horizon 2020, the Commission’s research and innovation programme and, as will be seen below, a very significant item in the EU budget.

3.4. The chapter therefore consists of three parts. Horizon 2020 performance is the subject of the first part. The remaining two parts deal with subjects we have examined in previous years: the extent to which performance is covered in the main annual planning and reporting documents prepared by the Commission’s directors-general and some of the main themes arising from our 2015 special reports including the follow-up of a sample of recommendations from previous special reports and, for the first time, responses from the Member States to a survey that we conducted on their follow up of our recommendations.

3.5. In the 2014 annual report, we reported that, based on information supplied by the Commission, the mid-term review of Europe 2020, which should have been available to the Council in March 2015, had been delayed until early 2016 (4). As of June 2016, the Commission informed us that the mid-term review had been finalised and that there would now be no formal mid-term review concluding document. As a result, we are not in a position to report in this chapter on how this review process was carried out. We may, however, return to this issue in future reports.

PART 1 — HORIZON 2020 (2014-2020)

3.6. There are three elements to our analysis of Horizon 2020. First we set out the main features of the programme and examine the strategic framework in which it operates, in particular its links to other EU strategies. The second element considers the advantages and potential risks involved in bringing research and innovation activity together. Finally we analyse the extent to which there is a focus on performance within Horizon 2020.

Section 1 — Links between Europe 2020, the new Commission priorities and Horizon 2020 do not always facilitate a focus on performance and need to be clarified

Europe 2020 and Horizon 2020

3.7. Europe 2020 is the European Union’s 10-year jobs and growth strategy. It was launched in 2010 to create the conditions for smart, sustainable and inclusive growth. Responsibility for delivering the Europe 2020 strategy is shared between the EU and its 28 Member States. Thus the strategy’s five headline targets (5) have an associated national target in each Member State, reflecting the different circumstances of each.

3.8. Horizon 2020 has a budget of around 75 billion euro for 2014-2020. It is the EU’s programme for supporting research and innovation, with the general objective to ‘contribute to building a society and an economy based on knowledge and innovation across the Union’ (6). It supports the Europe 2020 strategy and the EU objective of creating a European Research Area (ERA) (7). It brings together previously separate spending programmes for research and innovation (8).

3.9. Funding is provided mainly in form of grants, although access to other forms of finance, such as financial instruments, is also available. The Commission manages the majority of expenditure under Horizon 2020, either itself or through agencies and other bodies. Beneficiaries of Horizon 2020 are individuals, teams, universities, research institutes, SMEs and larger companies, from within and outside the EU. Many funded projects are collaborative.

3.10. Figure 3.2 shows the structure of Horizon 2020. The general objective is translated into three priorities (1. Excellent science, 2. Industrial leadership and 3. Societal challenges). These priorities are pursued through 14 ‘specific objectives’, complemented by broad lines of activities which provide their scientific and technological content. The general objective is also pursued through two specific objectives (‘Spreading excellence and widening participation’ and ‘Science with and for society’), while two additional organisations, the JRC and the EIT also contribute to the general objective and the priorities by following their own specific objectives. Horizon 2020 also includes additional research measures, such as ‘Fast-Track to Innovation’ to promote close-to-the-market innovation activities.

Figure 3.2 — The structure of Horizon 2020 Source: European Court of Auditors.

THE COURT’S OBSERVATIONS THE COMMISSION’S REPLIES

There are high-level links between Horizon 2020 and Europe 2020 but need to be enhanced to ensure that they facilitate a greater focus on performance

3.11. Under the legislation, Horizon 2020 has three performance indicators for assessing progress against its general objective:

(a) the research and development intensity indicator according to which investment should account for at least 3 % of GDP by 2020 — replicating one of Europe 2020’s five headline targets;

(b) an innovation output indicator;

(c) the share of researchers in the active population (9). This indicator has no direct link with Europe 2020 headline targets.

3.12. The 3 % indicator measures the input of funds in research and development. It was not designed to measure output, results or impact. Horizon 2020 adds for the first time in legislation the requirement that private — as opposed to public — investment in research and development should account for two-thirds of the total (10). Against these targets, in 2014 total such investment across Europe was just over 2 % of GDP, with the private sector accounting for around 64 % of this. Comparable figures for the USA were 2,8 % in 2012, 3,5 % for Japan and 4,2 % for South Korea (for both countries in 2013). 3.12. Whilst the 3 % R&D intensity indicator measures the input of funds in research and development, it also reflects the effectiveness of policy measures taken to stimulate a high quality science base in the Member States and to create better framework conditions for research and innovation in the private sector.

3.13. Horizon 2020 spending is one relatively small contribution to the target. It was always expected that Member States would provide the bulk of funds in this area. In 2014, the first year of implementation, Horizon 2020 spending by the EU institutions represented just under 10 % of total public budgets for R&D. We noted in last year’s annual report that meeting all national targets for research and development would lead to a total investment of 2,6 % of GDP by 2020, some way below the 3 % target (11). 3.13. In the area of research, as well as Horizon 2020, efforts are made to coordinate and improve Member States’ investment in research and innovation, and to develop framework conditions that allow private sector innovation to thrive (open innovation). This is crucial to making significant progress in closing the innovation divide between the European Union and its key competitors. Increasing impact and excellent science through openness (open science) and a digital single market, embedding digital into the grand societal challenges, implementing the research, innovation and competitiveness dimension of the Energy Union, together with a forward-looking climate-change policy, and translating Europe’s strengths in science and technology into a leading global voice (open to the world) are also key objectives of research policy. Research and innovation investments from the cohesion policy programmes for 2007-2013 also contributed to the progress towards the 3 % target. This contribution is continued for the 2014-2020 programming period thanks to the thematic concentration of ERDF and will strengthen its impact and leverage thanks to the ‘Smart Specialisation’ex ante conditionality for research and innovation. The contribution of Horizon 2020 to the achievement of the Europe 2020 strategy is reinforced by the interaction with other actions at EU and national level.

3.14. The second Horizon 2020 performance indicator, on innovation, is a composite output indicator. It was initially developed for the Europe 2020 flagship initiative ‘Innovation Union’ (12). It captures information on four elements: patents, employment in knowledge-intensive activities, competitiveness of knowledge-intensive goods and services, and employment in fast-growing firms of innovative sectors. The Composite Indicators Research Group (financed by the EU) has pointed to the fact that such indicators, based on sub-indicators with no common meaningful unit of measurement and with no obvious method for weighting, have pros and cons. In particular, while they illustrate the big picture, there is a risk that in isolation they can lead to simplistic policy conclusions; they should therefore be used in combination with the sub-indicators. The Commission reported on this indicator and the sub-indicators in March 2016 (13).

3.15. The innovation output indicator has the advantage that it focuses attention on establishing the right conditions for innovation and that it provides a measure for assessing progress in innovation at the level of the EU economy. However, it also has a number of shortcomings which limit its usefulness for monitoring the Horizon 2020 contribution towards the Europe 2020 strategy objectives: 3.15.

— the Commission has not yet set a quantified target to be reached by a defined date (14); — In order for a quantitative target to impact policymaking and stimulate progress, the target set must be consensual, justified on the basis of evidence and benchmarking, and easy to understand and communicate. This is more difficult to achieve in the case of composite indicators, such as the Innovation Output Indicator, as the summary scores in which they result are by nature abstract. Therefore, composite indicators typically do not come with target levels. Another example of this is the Summary Index of the Innovation Union Scoreboard, which also does not have a target level, but still clearly has an impact on policymaking of Member States. Furthermore, the available time series for some of the component indicators are still too short to establish trend lines and identify a meaningful EU target level (the component on high-growth enterprise was based on a new Eurostat data collection). In addition, setting an EU target level would require benchmarking against the performance non-EU countries, something not possible at the time the indicator was launched.

— the Commission has not translated it into national targets to reflect the individual situation of each Member State — as recommended by experts in the field for indicators measuring innovation (15). — The Research intensity target is set by Member State, and the Innovation Output Indicator complements it by providing the policy picture on innovation output. These different elements need to be seen together.

3.16. The specific objectives set by Horizon 2020 are the next level in the hierarchy of objectives where links are established to the Europe 2020 strategy. The largest element (38 %) of the Horizon 2020 budget is allocated to the priority ‘Societal challenges’ which is designed to address policy priorities in the Europe 2020 strategy.

3.17. In many cases, the description of the specific objectives refers to the flagship initiatives they are designed to support. However, although they are a useful indication of the generic relationship between Horizon 2020 and the Europe 2020 strategy, they are not further translated at lower levels into more specific and operational links that could serve as the basis for monitoring and reporting the contribution of Horizon 2020 to the individual flagship initiatives and thus the Europe 2020 strategy. 3.17. For the preparation of the budget, the contribution of Horizon 2020 to each flagship initiative is provided in the Horizon 2020 programme statement on an annual basis, following a methodology developed with the launch of the programme.

The links between the Commission’s 10 political priorities and Europe 2020/Horizon 2020 need further clarification

3.18. In 2013, the regulation establishing Horizon 2020 set priorities, general and specific objectives for the period 2014-2020. These priorities and objectives were designed to be aligned with, and to contribute to, the Europe 2020 strategy (16). Towards the end of 2014, the current Commission set 10 priorities for the Commission as a whole for 2014-2019 (17). While these priorities highlight some new areas (such as a free trade agreement with the United States), they also overlap significantly, as one would expect, with the existing Europe 2020 strategy, including its headline targets and flagship initiatives. For example, employment and climate change are issues which feature heavily in both. Despite the overlaps and complementarities, it has been difficult to establish the extent to which the 10 political priorities form a distinct new approach of the Commission or are intended as a new way for the Commission to put Europe 2020 into operation for its work. 3.18. The Europe 2020 strategy is the growth strategy for a decade for the EU and at EU level. In line with the Europe 2020 strategy, the 10 political priorities of the Commission provide a clear and focused agenda/strategy for the activities of the Commission towards Europe 2020 and they are entirely consistent with the overall Europe 2020 goals. At the same time, they provide greater clarity and focus in the Commission’s overall political strategy, taking into account recent developments. For example, since the establishment of the Europe 2020 strategy the EU has seen sovereign debt crises, increasing public sector deficits, war in Ukraine and Syria, a massive flow of migrants towards it, and the COP 21 climate agreement. These 10 political priorities have also been presented to the European Parliament, the European Council and the Council in the course of the appointment of the new Commission. The priorities take over, as targets and indicators, the targets and indicators of the Europe 2020 strategy.

3.19. It is normal for strategic priorities to alter, as circumstances change. However, as pointed out by the Commission’s internal audit service, although there is common ground between the two strategies, the Commission has, so far, not mapped the correspondence between them. In our view, since the relationship between the two strategies has not been clarified, they coexist and are used for different purposes: 3.19. The Commission considers that its priorities provide a clear and focused policy agenda for the Commission that is consistent with the Europe 2020 strategy for the European Union as a whole. As such, they are naturally used for different purposes. However, this does not mean that they are in any way inconsistent or incoherent, they are entirely complementary. Please refer also to the Commission reply to paragraph 3.18.

— The allocation of the EU budget to spending programmes (such as Horizon 2020) is based on, among other things, their contribution to the Europe 2020 headline targets and flagship initiatives. The monitoring and evaluation of spending programmes also builds on objectives designed to reflect the Europe 2020 strategy. — In relation to Horizon 2020 this was implemented in the first work programmes, for 2014-2015. Over the remainder of the programme the work programmes, and the budget for particular elements of the work programme, will be increasingly influenced by the political priorities set out by the current Commission in response to the emerging political, economic and technological context.

— The Commission’s annual work programme now focuses on the 10 political priorities. Starting from 2016, the Commission has also introduced a new method of strategic planning, which requires DGs to prepare 5-year strategic plans, also based on the 10 political priorities. The 5-year strategic plans will become the starting point for drafting annual management plans (MPs) for carrying out the DGs’ tasks (including the implementation of spending programmes such as Horizon 2020). Since the reporting in annual activity reports is strongly linked to the MPs, annual activity reports from 2016 onwards can also be expected to reflect primarily the 10 political priorities. — The Commission’s annual work programme includes the actions that the Commission will take to help deliver on the political priorities and contribute to the achievement of the Europe 2020 strategy. The new multiannual Strategic Plans developed for each Commission department are built around the 10 political priorities, expressed as general objectives. As part of this exercise, the DGs were instructed to explain how their activities contribute to the Europe 2020 strategy. To facilitate this, the headline targets of the Europe 2020 strategy appear as impact indicators linked to the relevant general objectives. The accompanying annual management plans set out the planned outputs for each department and how these will contribute to the objectives. Commission departments will report on their achievements through their annual activity reports, which will reflect the structure of the strategic and management plans. However, the Commission will continue to report on progress towards the Europe 2020 targets as part of the European Semester process and on the contribution of the budget to Europe 2020 in the annual management and performance report.

3.20. For Horizon 2020, this leads to a situation where the legal framework and the budget allocation for the programme reflect the Europe 2020 strategy, while those parts of the Commission implementing Horizon 2020 have refocused strategic planning and management arrangements on the 10 political priorities. For example, the key priorities to be developed in the Horizon 2020 work programmes 2016-2017 are based on seven out of the 10 policy areas on the agenda of the new Commission. Under these circumstances, it is unclear what strategic objectives Horizon 2020 should be targeting. DG RTD identified four political priorities where the DG can make a contribution (18). The pursuit of these priorities, however, will require a refocusing of effort against the original targeting as set out in the Horizon 2020 legislation. 3.20. The Commission considers that there is no lack of clarity about which strategic objectives Horizon 2020 should be targeting. These objectives are the Europe 2020 objectives, as set out in the legal base. The 10 political priorities are used to consider how the Commission can best assist in reaching the Europe 2020 priorities. For example, the delivery of the digital single market (DSM) strategy will lead to updated priorities in the work programme 2017-2018 of the ICT part of Horizon 2020 in line with both Europe 2020 and the digital single market Commission priority, within the flexibility available in Horizon 2020. The respective set of measures and initiatives of 18 April 2016 (DSM package) already demonstrates this alignment of Horizon 2020 and the DSM priority.

3.21. In last year’s annual report we pointed to the weak alignment between the Europe 2020 and the EU’s seven year budgetary cycle (19). The same complication may affect the 10 political priorities, as the mandate of the current Commission ends in 2019, a year before the end date of Europe 2020 and Horizon 2020. 3.21. The 2007-2013 MFF period may not have fully coincided with the period covered by the Europe 2020 strategy. However, the Commission maintains its position formulated in reply to paragraphs 3.7-3.11, 3.87 of the Court’s annual report for 2014, that this does not imply a lack of alignment with the overall strategy. Moreover, the 2014-2020 MFF period falls entirely within, and has the same end date, as the current Europe 2020 strategy period. The 2014-2020 MFF has been designed with a view to concentrate on delivering on the Europe 2020 strategy. The Commission is monitoring and reporting on the contribution of the EU budget to the EU’s overall objectives and to the Europe 2020 strategy in particular. The current Multiannual Financial Framework, Horizon 2020 and the Europe 2020 strategy are fully aligned and have a common end date of 2020. These frameworks are not affected by the expiry of the mandate of the current Commission in 2019.

Section 2 — Synergies and complementarities with ESI funds and national programmes

Limited information on synergies with ESI funds

3.22. One of the main objectives of the Europe 2020 strategy and its Innovation Union is the completion of the ERA in which researchers, scientific knowledge and technology circulate freely. The EU budget contributes to the ERA objective through Horizon 2020 and the ESI funds. In this context, and in line with the legislation, the Commission envisaged a system for encouraging and tracking synergies between Horizon 2020 and the ESI funds. The classification of Horizon 2020 projects according to ESI thematic objectives, together with the introduction of specific references to potential synergies at their planning and reporting stages would provide the basis for such a system. However, in the first Horizon 2020 monitoring report, information on the use and impact of synergies between Horizon 2020 and the ESI funds is limited. The Commission plans to analyse the impact and synergies between Horizon 2020 and the ESIF in the context of the Horizon 2020 interim evaluation. 3.22. The Commission has created the regulatory conditions that can enable synergies between the funds by increased coordination and harmonisation of the relevant legal frameworks. However, the implementation of synergies needs a continued effort both on the side of the bodies in charge of Horizon 2020 and of the authorities of the Member States and regions. Significant efforts were undertaken in Member States and regions, in particular in the framework of the development of smart specialisation strategies to develop policy measures that allow to implement the different synergy types identified by the Commission’s guidance and to focus R&I efforts on fields where the Member State or region has the biggest chances for excellence in terms of science and business competitiveness. These fields also offer the biggest chance to be successful in Horizon 2020 calls. Also on the side of Horizon 2020 management, extensive action is being undertaken to promote and develop synergies, for example through the set-up of memoranda of understanding between joint undertakings or the seal of excellence initiative, launched at the initiative by Commissioners Moedas and Creţu in October 2015. A number of regions and Member States are already involved in this initiative. The Commission also implements the ‘Stairway to Excellence’ initiative helping the EU-13 Member States to improve their access to Horizon 2020.

Examples of complementarities between national and EU research programmes

3.23. The Horizon 2020 legislation recognises the importance of building ‘synergies and complementarities between national and European research and innovation programmes’ (20). Figure 3.3 below includes key messages from our cooperation (21) with the Supreme Audit Institutions (SAIs) of Bulgaria and Portugal: while there are some areas in which national and EU research programmes are complementary, our partners also found some problems at the national level.

Figure 3.3 — Examples of complementarities with national programmes including some identified weaknesses at national level Figure 3.3 — Examples of complementarities with national programmes including some identified weaknesses at national level

Bulgaria (according to an audit by the Bulgarian National Audit Office (BNAO) published in July 2016) Bulgaria (according to an audit by the Bulgarian National Audit Office (BNAO) published in July 2016)

The relation between the EU priorities in the context of the Europe 2020 strategy for smart, sustainable and inclusive growth, including Horizon 2020, and the national priorities of Bulgaria, is articulated through eight priorities set in the National Development Programme Bulgaria 2020 (NDP). Three of these priorities are related to science and innovation:

— Improving the access to and enhancing the quality of education and training and the quality characteristics of the workforce.

— Support for innovation and investment activities to increase the competitiveness of the economy.

— Energy security and increased resource efficiency, particularly for SMEs.

In addition to the NDP, which is an integrated document for the socio-economic development of Bulgaria up to 2020, the objectives under Horizon 2020 are also synchronised and broken down into operational objectives and plans in:

— the National Strategy of Scientific Research (NSSR) to 2020, which is designed to support scientific development in Bulgaria and make it into a factor for the development of an economy based on knowledge and innovation activities,

— the Innovation Strategy for Smart Specialization (ISSS), that has the strategic objective for Bulgaria to move from the group of ‘modest innovators’ to the group of ‘moderate innovators’ by 2020.

The national strategies set a national objective to increase the share of expenditure on research and development activity to 1,5 % of GDP by 2020. To achieve the national objective measures and activities are envisaged in the Action Plans implementing the national strategic documents. However, the BNAO identified that neither quantitative indicators for results, nor target values per year for activities related to Horizon 2020 are included in the action plans for the NDP. Neither were quantitative indicators per activity included in the action plan for the NSSR. The BNAO has also identified some issues regarding coordination and interaction between all the participants engaged in Horizon 2020 at the national level, and that improving the interaction between all the participants would stimulate Bulgaria’s participation in the programme. The Commission supports Bulgaria actively both via the ‘Stairway to Excellence’ pilot (see above) and the new ‘lagging regions’ initiative (1) in order to improve its innovation eco-system and participation in Horizon 2020 via the implementation and evolution of the Bulgarian smart specialisation strategy. The European Commission presented in October 2015 key policy recommendations to help modernise the research and innovation sectors in Bulgaria. This was the first time that an EU Member State made voluntary use of the Horizon 2020 Policy Support Facility (PSF), a new instrument that gives governments practical support to identify, implement and evaluate those reforms needed to enhance the quality of their public research and innovation structures.

Portugal (according to specific audit work by the Portuguese Tribunal de Contas during the first quarter of 2016)

The main national strategic document in the area of research and innovation is the Research and Innovation Strategy for Smart Specialization (RISSS) that has a 2014-2020 timeframe, matching the Horizon 2020 timeframe and Europe 2020 end date. One of its chapters includes links between Europe 2020 and the Portugal 2020 partnership agreement, signed between Portugal and the Commission, in the area of R&D and innovation, mobilizing funds from both cohesion policies and Horizon 2020.

The Portuguese Tribunal de Contas noted that the RISSS included two scenarios for the evolution of the indicator on the proportion of GDP invested in research and development: a ‘conservative’ one of 2 % achieved by 2020, and a ‘change’ one with 2,7 % by 2020. It also found that, taking account of Europe 2020, Horizon 2020 indicators and the Innovation Union indicators, some issues are not tackled by the national strategy indicators, such as the researchers’ gender, the contribution that is sustainable development and climate change related, and the participation of SMEs.

The Tribunal de Contas also noted that there are similar indicators at national and European level, such as those dealing with scientific publications or innovative products or processes, that cannot be compared, for instance because different denominators are used.

Section 3 — The mechanics of Horizon 2020 do not consistently drive a focus on performance

3.24. Measuring performance in the area of research is inherently difficult. The results and impacts of research by their nature are difficult to predict, generally rather long-term and often do not follow a linear path. However, the EU still needs to obtain reliable information on scientific progress, productivity and quality for the research it finances, in order to measure scientific performance for good decision making. This need is recognised in the Horizon 2020 framework regulation which states that ‘effective performance management, including evaluation and monitoring, requires the development of specific performance indicators […] which are relevant to the appropriate hierarchy of objectives and activities’ (22).

3.25. Against this background, this section examines whether:

(a) there is a focus on performance, including a clear and operational link to Europe 2020, in the grant management process of Horizon 2020;

(b) the design of the monitoring and reporting system of Horizon 2020 enables the Commission to monitor and report on performance; and

(c) important basic concepts of performance monitoring and reporting are used in a consistent and correct manner by the Commission.

The grant management processes are not fully designed to focus on performance

3.26. To assess whether the Horizon 2020 grant management processes included an appropriate focus on performance, particularly in its links to Europe 2020, we examined the design of the Horizon 2020 legal framework, the focus on performance at the level of work programme and calls and, for each of the five work programmes we sampled (23), one successful proposal by beneficiaries.

The legal framework of Horizon 2020 introduces several important elements for performance management but there are some weaknesses in design, particularly as regards the use of performance indicators

3.27. The objectives and indicators agreed by the legislative authority for Horizon 2020 represent a real improvement on the previous framework programmes. As we have observed in the past, objectives defined in legislation may be at too general and high a level for operational use by management (24). This is also the case for these high-level Horizon 2020 objectives — even those referred to as specific objectives in the regulation, most of which are not actually formulated as objectives but rather as challenges and/or areas. For example, for the priority ‘Societal challenges’ two specific objectives are: ‘health, demographic change and well-being’; and ‘secure clean and efficient energy’. The extent to which these high-level objectives are successfully translated into lower-level objectives by management is examined in the next section. 3.27-3.28. For Horizon 2020, the inclusion of objectives in the legal text, for the first time, was an undoubted step forward. It did lead to an extensive debate with the legislative authority about the objectives of the programme, and the indicators to be monitored.

3.28. As well as specifying the objectives of the programme, the Horizon 2020 legislation also includes three performance indicators for assessing progress against the general objective, set out in paragraph 3.10. At the next level down of the legal framework, the specific programme (25) introduces further elements important for performance management. It lists 18 key indicators for assessing results and impacts of the specific objectives of the Horizon 2020 programme (26). In addition, the Commission has identified a further five key performance indicators (KPIs) giving 23 in all (27). The inclusion of performance indicators in the legislation is a positive step. However, we found a number of shortcomings with regard to the programme indicators, set out in the following paragraphs.

3.29. Of these 23 key performance indicators, more than half measure input or output rather than results and impacts (see also Figure 3.4 ). Furthermore there are ambiguities in the formulation of some indicators, which make their interpretation difficult without further clarification. Examples include the ‘Number of institutional change actions promoted by the programme’ or the ‘Number of occurrences of tangible specific impacts on European policies resulting from technical and scientific support provided by the Joint Research Centre’ (28). 3.29. Ambiguities in formulation may always exist, especially in legislation, so there are often further clarifications. For example, the ‘Number of occurrences of tangible specific impacts on European policies resulting from technical and scientific support provided by the Joint Research Centre’ is further defined in an indicator fiche, which also includes a reference to the methodology for measuring the indicator.

3.30. For a third of the Horizon 2020 key performance indicators, the Commission did not set either baselines, milestones, or targets on the grounds that these indicators represent a ‘new approach’, and therefore no targets can be set before the first results are available, due at the beginning of 2018. Only five of the 23 Horizon 2020 key performance indicators have both a baseline and a target. In our view, 2018 is much too late to be useful for a programme lasting from 2014 to 2020. Where targets do exist, they are sometimes unambitious. In some cases, targets are set below the level of results achieved under FP7. In addition, we found five Horizon 2020 indicators (29) in the 2015 programme statements where the target was below the baseline. 3.30. While the Legislative Financial Statement accompanying the Commission’s proposal for Horizon 2020 generally provided baselines and targets for indicators, a number of new indicators were included in the legislative process, and for these there was no baseline on which to consider a reasonable target. Setting a target without adequate evidence may lead to them lacking relevance and credibility (two of the RACER criteria). Horizon 2020 targets that are below the FP7 results are explained. For example, for the number of researchers who have access to research infrastructures through union support has been reduced because ‘priority will be given to new emerging infrastructures and the targeting of new communities (starting communities) whose infrastructures are usually not able to provide as large an access as advanced communities’ (annual report of DG RTD, page 33). In many areas targets are ambitious and increased from FP7 — for example SME involvement, climate change related expenditure, expenditure on renewable energy, time taken to sign grants, to give some examples.

3.31. For several indicators, no meaningful data will be available before 2018/2019 for reporting. For a quarter of the Horizon 2020 key performance indicators, the Commission considers that it is not meaningful to report results before 2019, because a critical mass of finished projects will not be reached before then. While these indicators might be useful for the ex post evaluation of Horizon 2020, they are not suitable for interim evaluations or for annual monitoring purposes before 2018. This is confirmed in the Horizon 2020 Monitoring Report 2014. 3.31. Compared to other funding areas, there can be a considerable time lag between the moment research funds are disbursed and the moment research produces outputs and the moment when these outputs produce results (during and after the project’s lifetime). For example, at the time of the ex post evaluation of FP7 published in January 2016 (hence, 2 years after the end of FP7), more than half of FP7 projects were still ongoing. For this reason, the Commission will continue reporting on FP7 results in a dedicated section of the monitoring reports of Horizon 2020. Moreover, the interim evaluation of Horizon 2020 will include an assessment of the longer term impacts of previous framework programmes. This approach will continue over time mitigating this problem.

3.32. A practical example of the problems caused by the poor use of indicators is provided in our special report on the EIT. We noted that the organisation’s core indicators continued to focus on input or output rather than results or impacts, and concluded that it was difficult to assess the performance of the EIT since it started operations in 2010. There was a lack of robust and relevant reporting and monitoring processes and few impacts were visible (30). 3.32. As stated in the reply of the Commission and EIT to special report No 4/2016, the EIT performs comprehensive analyses to compare the Knowledge and Innovation Communities (KICs) based on several factors, not simply on the basis of the core KPIs. The implementation of KICs’ business plans is assessed both through quantitative performance indicators and through qualitative reviews done by external experts and the EIT’s staff. More widely, the EIT also conducts on-site monitoring and topical reviews of the KICs’ activities. These tools, taken together, provide a very informative picture of the differences between the KICs.

There is insufficient focus on performance at the level of work programmes and calls

3.33. The work programmes and associated calls are a key driver in the way that Horizon 2020 operates. They provide an opportunity for the Commission to translate the high-level general and specific objectives of Horizon 2020 into lower level objectives useful for operational performance management purposes.

3.34. Based on the specific programme, the Commission prepares 2-year work programmes that develop the specific objectives and expected impact, and set out funding opportunities through calls for proposals: 15 billion euro for 2014-2015 and 14,5 billion euro for 2016-2017. The ‘main’ Horizon 2020 work programme dedicates one part to each specific objective of the specific programme. It is complemented by separate work programmes for the ERC (an annual programme), the JRC (biennial programmes), and the EIT (a triennial programme, broken down annually). 3.34. Since the observation of the Court the Commission has introduced a strategic planning process that went on before the 2016-2017 work programmes were finalised. This is a strategic examination of priorities and needs, carried out with Member States and other stakeholders, to ensure the link between the Horizon 2020 specific objectives and the work programmes implementing them.

3.35. The legislation for Horizon 2020 was adopted shortly before the implementation of the programme started, limiting the scope for strategic planning. The Europe 2020 strategy, the ‘Innovation Union’ and other flagship initiatives are referred to in the overall strategic framework for the work programme 2014-2015, but the references remain generic, unsystematic and do not clarify the specific contribution of the individual work programmes to particular strategic objectives. For 2016-2017, the strategic programming process improved and was documented via an overarching strategic planning document and scoping papers. 3.35. The preparation of the Horizon 2020 legislation was accompanied by the parallel preparation of the work programmes 2014-2015. There was a clear link of each work programme part to the Horizon 2020 specific objectives.

3.36. For all the work programmes we examined, the objectives are consistent with the specific objectives as defined in the specific programme. However, the objectives in the sampled work programmes do not translate the high-level objectives from the legislation into operational ones which can be monitored and reported on: three out of five work programme objectives use exactly the same wording as is in the legislation, and none fulfil the SMART (31) criteria. The objectives are not accompanied by indicators with targets and milestones. Operational objectives and relevant indicators would facilitate the required focus on performance or monitoring and reporting on performance at the work programme level. 3.36-3.38. It is accepted that the ‘expected impact’ established in the work programmes can be improved, and this has already started in the 2016-2017 work programmes. However, the primary focus of the work programme is to provide the information necessary for applicants to prepare high quality proposals. Work programmes do not need to identify their contribution to Europe 2020 objectives. They should rather be linked to the specific objectives of Horizon 2020, and these objectives have been formulated to comply with Europe 2020. Performance will firstly be assessed through the indicators set out in the legislation. This is made clear in Article 5 of the Horizon 2020 legislation. ‘The first set of relevant performance indicators for assessing progress against the general objective is set out in [the Annex to the Regulation]’. This will be the primary information on which the assessment of performance will be established.

3.37. Each work programme contains calls for proposals corresponding to the broad lines of activities of the specific objective (representing the scientific and technological content). The sampled calls for proposals identify objectives which are in line with those in the work programme and consistent with specific broad lines of activities, where applicable (32). Although overall the objectives of the calls are more specific than the ones in the work programme, they do not meet all SMART criteria. Beside objectives, each call also includes an expected impact section. However the expected impacts are rarely quantified and generally do not include indicators. It will thus be difficult for the Commission to measure the success of the calls in the absence of operational objectives and indicators with targets.

3.38. The sampled work programmes do not identify their long-term contribution to the strategic objectives even though this should be explained according to guidance on work programmes. None of the examples we analysed explain how the work programme contributes towards the achievement of specific Europe 2020 objectives; in one case, the objective of the work programme links the actions supported to one of the flagship initiatives but does not specify exactly how.

THE COURT’S OBSERVATIONS

There is a focus on performance in proposals, evaluations and grant agreements but there are weaknesses that can lead to problems when assessing performance

3.39. Under the calls, applicants submit proposals for funding. The Commission evaluates the proposals and for those it chooses to fund the Commission signs grant agreements with beneficiaries. In this section we assess the focus on performance for these stages of the management process.

Evaluation of proposals

3.40. Proposals are normally evaluated by experts, on behalf of the Commission, based on their excellence, impact and the quality and efficiency of their implementation (33). With the exception of the ERC, which uses only the excellence criterion, the beneficiaries’ proposals we examined all included sections structured around these three evaluation criteria, with the weighting for impact at least as high as the other two criteria. However, the quality of this evaluation depends on the quality of the expected impacts set out in the calls published by the Commission.

3.41. For the excellence criterion, templates included in proposals indicate how objectives should be defined. Three out of the five proposal templates sampled required applicants to define SMART objectives and explain how they relate to the topic of the relevant work programme. The objectives of these three proposals are generally SMART. However, for the other two proposals, related to the ERC and Marie Curie, the sampled proposals did not include SMART objectives as there was no such requirement.

THE COURT’S OBSERVATIONS THE COMMISSION’S REPLIES

Grant agreements

3.42. Grant agreements are the main way in which Horizon 2020 funds are allocated to deliver the programme objectives. The sampled grant agreements used the same objectives as were in the proposals.

3.43. Grant agreements include a section on ‘background and results’. These results mean ‘any tangible and intangible output of the action’ as defined in the rules of participation (34). There is an important provision in the grant agreements which requires beneficiaries to report to the Commission information for the aggregation of the Horizon 2020 KPIs; this level is therefore an important source of all performance information for Horizon 2020. We have pointed to problems assessing performance at the higher level (work programmes and calls) in the previous paragraphs. 3.43. See Commission’s replies to paragraphs 3.36 to 3.38. Information on the indicators established in the legal base can and will be collected to allow an assessment of performance.

The design of the monitoring and reporting system of Horizon 2020 limits the Commission’s ability to properly monitor and report on performance

3.44. This section focuses on the monitoring and reporting system for Horizon 2020. We examined whether:

(a) the current setup enables the Commission to monitor and report separately the spending on research and development as opposed to innovation;

(b) the contribution of Horizon 2020 to Europe 2020 can be reported on;

(c) the Commission will be in a position to evaluate the programme in the course of its life.

Spending on research is not monitored separately from that on innovation

3.45. Horizon 2020 brings together previously separate funding instruments for research and innovation in a common strategic framework (35). In line with this approach, the innovation-related parts of the Competitiveness and Innovation Framework Programme (36) and the EIT were brought together under the umbrella of Horizon 2020.

3.46. We pointed out in the past that the proposed legislation for Horizon 2020 did not provide a clear definition of innovation and that without some limits on the scope of fundable innovation activities there was a risk that a wide range of innovation projects would dilute funding (37). In our view this risk persists. The framework programme and the specific programme for Horizon 2020 do not provide clear criteria to distinguish between ‘research and development’ and ‘innovation’ (38) or to monitor and report separately on the performance under these headings. 3.46-3.47. One of the major aims of Horizon 2020, set out from the beginning of discussions on the framework programme, was to narrow the divide between research and innovation. The legislation gives no separate definition of research and innovation instead stating that: ‘“Research and innovation activities” mean the whole spectrum of activities of research, technological development, demonstration and innovation, including the promotion of cooperation with third countries and international organisations, the dissemination and optimisation of results and the stimulation of high quality training and mobility of researchers in the Union’. The Commission considers that distinguishing between research and innovation activities is not required, has no clear added value, and risks to run counter to the efforts to narrow the gap between research and innovation in Europe.

THE COURT’S OBSERVATIONS

3.47. The EU has adopted the 3 % headline target for the research and development intensity indicator. Not all of the innovation activity funded by Horizon 2020 falls within the definition of research and development for the purpose of this target (39). As a matter of fact, there are activities within Horizon 2020 that aim specifically at fostering innovation (40) as opposed to research and development and, therefore, are not captured by the 3 % target.

The contribution of Horizon 2020 to Europe 2020 whilst established in the budget will not be reported on in meaningful way at least until 2018

3.48. The programme statements, used in the budgetary procedure to justify operational expenditure, provide information on the links between spending programmes and the Europe 2020 strategy by signalling the headline targets pursued and by estimating the financial contribution of programmes to the flagship initiatives. Therefore, the financing of the Europe 2020 strategy from the EU budget is calculated based on the estimated financial contribution of spending programmes to the flagship initiatives in the programme statements.

THE COURT’S OBSERVATIONS THE COMMISSION’S REPLIES

3.49. In terms of the budgetary process, through the programme statements Horizon 2020 links over 90 % of its 2015 budget to the different flagship initiatives. Almost half of this sum (4,3 billion out of 9,5 billion euro) was allocated to actions related to flagship initiative ‘Innovation Union’.

3.50. While links between Horizon 2020 and Europe 2020 are established for the budgetary procedure, they are not reflected in the performance reporting on Horizon 2020. The Horizon 2020 legislation requires monitoring to include information on cross-cutting topics, one of which is the contribution to all relevant Europe 2020 flagship initiatives (41). However, the Horizon 2020 annual monitoring report 2014 (published in April 2016) does not address the contribution of Horizon 2020 to the relevant flagship initiatives. The links of Horizon 2020 and Europe 2020 will only be assessed in the interim evaluation report of Horizon 2020, expected in 2017. And none of the Commission’s evaluation reports required under Article 318 of the TFEU over the past years have reported on this contribution in a useful way. 3.50. The links between Horizon 2020 and the Europe 2020 strategy will be assessed in depth in the interim evaluation of Horizon 2020. The interim evaluation of Horizon 2020 will also assess the leverage issue. The Commission considers that Article 31 of the Regulation establishing Horizon 2020, which establishes the scope of the annual monitoring report, does not require that it addresses the contribution of Horizon 2020 to the relevant flagship initiatives of the Europe 2020 strategy. The monitoring report is a report on the implementation of Horizon 2020 only. According to Article 32 of the Regulation establishing Horizon 2020, the contribution of Horizon 2020 to the objectives of the Europe 2020 strategy will be taken into account in the interim evaluation of Horizon 2020. The Commission’s evaluation reports under Article 318 of the TFEU can only report on the contribution of Horizon 2020 to Europe 2020 when this information is available after the interim evaluation.

Some inherent difficulties encountered in the ex post evaluation of FP7 may persist also for Horizon 2020

3.51. There are inherent difficulties in terms of timing concerning evaluations of research framework programmes. The framework programme for Horizon 2020 states that evaluations should be carried out ‘in a sufficiently timely manner to feed into the decision-making process’. This requirement is difficult to fulfil in the area of research, where results and impacts might take many years to appear. Nonetheless, the legislation sets deadlines for different evaluations: an interim evaluation is required typically in the fourth year of the 7-year framework programme (2017 for Horizon 2020), and an ex post evaluation is requested 2-3 years following programme end (2023). While these deadlines are designed to support the decision-making process, the risk is that they lead to evaluations which do not adequately capture the results and impacts of programmes. 3.51-3.53. The Commission has recognised the inherent difficulty of providing meaningful evaluations within a fixed 7-year timeframe for each framework programme. This is why the Horizon 2020 interim evaluation will include an updated assessment of the performance of FP7. This will mitigate the problems identified. The Commission also recognises the need to balance the need to provide timely inputs into new legislation against the needs to evaluate the longer term effects of the added-value of research spending. This is reflected in the legislation. The Commission agrees that an interim evaluation of a research framework programme will face some inherent difficulties both in terms of outputs and in terms of data availability. However, leaving it until later may delay the learning of important lessons that can certainly be assessed at the stage of the interim evaluation. This is recognised in the legislation, as many of the specific elements in Article 32(3) that the interim evaluation is required to address can be assessed at a relatively early stage of the programme, for example assessing: — the participation of participants that have at their disposal high-end research infrastructures or have a history of using full-costing in the seventh framework programme, — the simplification for participants that have at their disposal high-end research infrastructures or have a history of using full-costing in the seventh framework programme, — extent of use of the additional remuneration to personnel as referred to in Article 27 of Regulation (EU) No 1290/2013. The Commission considers therefore that the legislation, and evaluation practices, mitigates as far as possible the inherent difficulties recognised by the Court.

3.52. The ex post evaluation of FP7 (2007-2013) was finalized towards the end of 2015, when more than half of FP7 projects were still ongoing and two years after the adoption of the successor programme, Horizon 2020. This evaluation was affected by delays in many FP7 projects and because the beneficiaries’ obligation to report stopped when the projects ended. Furthermore, the data collection for some FP7 performance indicators ceased in 2013, when the seventh framework period ended.

3.53. For Horizon 2020, the interim evaluation due to be undertaken in 2017 will face constraints because no meaningful results will be available for several performance indicators (see paragraph 3.31 above). And evaluation in general for Horizon 2020 is likely to be affected by the shortcomings identified in reporting results for calls and work programmes (where the data collected from projects cannot be aggregated to give feedback on calls and work programmes — see paragraphs 3.33 to 3.38); and by the difficulty the Commission faces in collecting performance information from the beneficiaries after the closure of projects. The Commission may carry out evaluations of the impact after the projects end, but in the cases we examined we found no requirements for beneficiaries to report after the closure of projects. 3.53. The Commission can aggregate information to assess the 23 indicators set out in the legal base, and this is the cornerstone of the performance framework, as set out in Article 5 of the Horizon 2020 Regulation: ‘The first set of relevant performance indicators for assessing progress against the general objective is set out in [the Annex to the legislation]’.

The Commission does not always use key performance concepts consistently, with the risk that the performance management system is undermined

3.54. Figure 3.1 sets out the various ways in which EU action can lead to benefits: ‘output’, ‘result’ and ‘impact’. ‘Outcome’ includes both results and impacts. These terms are used as part of the Commission’s performance monitoring and reporting system to define objectives and indicators. Any inconsistencies or weaknesses in the way that the terms are used risks undermining a performance management system.

3.55. The Commission broadly uses these terms in line with these definitions (42). However, in Horizon 2020, the rules for participation by beneficiaries define results as ‘any tangible or intangible output of the action, such as data, knowledge or information that is generated in the action’. This definition would normally be applied to ‘output’ rather than ‘result’. 3.55-3.57. The adoption of the Horizon 2020 Regulations (December 2013) preceded the better regulation guidelines (May 2015). This has led to some inconsistencies, as noted by the Court. Since the adoption of the better regulation guidance, the better regulation terminology is consistently used in the monitoring and evaluation of the performance of Horizon 2020.

3.56. Another inconsistency in the application of these terms in Horizon 2020 is that, under the legislation, the individual work programmes which drive Horizon 2020 should include ‘expected results’ (43) whereas the criterion used in the rules of participation (44) is ‘impacts’. However, the terminology used by the Commission in work programmes and calls for proposals is ‘expected impacts’.

3.57. Results measure the immediate effects directly attributable to an action, and are thus much more suitable as criteria for selecting projects than impacts, which are longer term and may include elements which cannot be addressed at project level.

3.58. Our sample of five ongoing Horizon 2020 projects shows that the notion of ‘expected impact’ was interpreted differently under the various calls. One beneficiary, for the SME scheme, was required to submit a market analysis and a business plan. These documents were part of the proposal and analysed by the evaluators for the impact criterion. Another beneficiary provided a brief qualitative description of the link between the project and the various expected impacts, such as ‘Increased innovation opportunities for service providers, including SMEs and public administrations.’ A third beneficiary described the benefits of a new treatment protocol that would become available as a result of the project, and confirmed that the results should contribute to the impact defined in the relevant call. The diverse interpretation of ‘expected impact’ in calls, and the different ways beneficiaries found to justify their contribution, poses the risk of inadequately assessing project performance in relation to the overall goals of calls and work programmes. 3.58. The need to strengthen the expected impact statements was identified as an issue for the second work programme and the existing guidance was improved. The quality of the evaluation process will still ensure that best projects are selected. Regarding the beneficiary that was required to submit a market analysis and a business plan, this is for the SME scheme and a market analysis and business plan are conditions for the grant. The emphasis of Horizon 2020 is on innovation and longer term impacts. This is why ‘expected impact’ has been used in the calls, and is assessed during evaluations. The use of the word ‘results’ in the legislation arises because the Horizon 2020 legislation preceded the better regulation guidelines. The contribution of projects in different areas is assessed using a range of indicators set out in the Horizon 2020 legal acts and other documents. And indicators are only one source of information. For the monitoring and evaluation of performance, the Commission uses a wide range of data to assess this (e.g. through external studies, surveys, case studies, counterfactual analysis etc.).

3.59. Another problem stemming from the Commission’s lack of clarity in this area is that some of the indicators described result or impact indicators actually deal with input and output therefore limiting the Commission’s ability to assess performance. Some examples of how these risks have materialised are in Figure 3.4 . 3.59. The indicators set out in the Regulation cover input, output, results and impacts. The Commission considers that monitoring of these indicators, backed up by evaluation, will provide a good basis on which to assess performance.

Figure 3.4 — Inconsistent use of indicators As paragraph 3.28 notes, there are 23 Horizon 2020 key performance indicators ‘for assessing the results and impacts of the specific objectives of Horizon 2020’. Seven of these indicators are linked to publications, such as ‘Leadership in enabling and industrial technologies (LEIT) — Number of joint public-private publications’. Achieving outputs such as this may be a valid objective of the programme, but such publication indicators do not measure either results or impacts. In addition, there are four other indicators that are linked to input rather than result or impact. Examples include ‘Access to risk finance — Total investments mobilised via debt financing and venture capital investments’ and the ‘Percentage of the overall Energy challenge funds allocated to the following research activities: renewable energy, end user energy-efficiency, smart grids and energy storage activities’. We also reported on this issue in chapter 3 of the last year’s annual report, in the context of the ESI funds (45). Figure 3.4 — Inconsistent use of indicators The Commission’s better regulation guidelines have established a standard definition of these terms and these definitions will now be applied consistently. While publications in themselves are outputs, the Commission considers that in certain cases they can be result indicators for example, publications in peer reviewed journals. The Commission considers that its performance framework, and especially the indicators used for Horizon 2020, provide an appropriate balance between different types of indicators. Within this balance, a certain number of input indicators is acceptable, to ensure that different priorities are adequately assessed.

PART 2 — PERFORMANCE PLANNING AND REPORTING AT DIRECTORATE-GENERAL LEVEL: THE ‘NATURAL RESOURCES’ FAMILY

3.60. This part of the chapter covers performance reporting at the level of Commission DGs. We reviewed the 2015 programme statements (46), management plans (47) and annual activity reports (48) of four DGs (49), based on requirements set out in the Financial Regulation, the Commission's internal control standards and its internal instructions relating to MPs and AARs.

3.61. According to the Commission’s 2015 MP instructions, DGs should establish general objectives with impact indicators (long-term) and specific objectives with result indicators (short/medium term). According to the Financial Regulation, DGs’ objectives should be SMART (50).

3.62. In line with our observations in previous years in relation to other DGs, we found that, for all four DGs reviewed, many of the objectives (general and specific) used in the MPs and AARs were taken directly from policy or legislative documents and were thus at too high a level to be useful as management instruments (see Figure 3.5 ). 3.62. Since the Court’s observation in relation to the management plans for 2015, the Commission has introduced changes to the SPP cycle notably to provide a clearer framework for the Commission’s accountability and for its services. With the new approach, the planning documents as from 2016 are meant to be more streamlined and focused on the priorities of the Commission and the competencies of the DGs, helping them to establish priorities and tell a coherent performance story with a clear distinction between attribution (for results and outputs that are a direct consequence of the Commission services’ activities) and contribution (where the Commission services’ activities play a part in bringing about a given result or an impact) for EU high level policy and spending programmes’ results. The strategic plan introduces a new approach shifting the focus from short-term planning to a long-term, strategic perspective helping the DGs to align their specific objectives with the long-term objectives defined at Commission. The operational planned outputs for each directorate-general for the year ahead are listed in the Management Plan.

Figure 3.5 — Examples of objectives not fit for management purposes DG CLIMA: The objective ‘To secure investment for climate related issues’ is not specific as it does not indicate the volume of investment to be secured, and it is not time-bound as there is no deadline to achieve it. DG ENV: ‘To turn the EU into a resource-efficient, green and competitive economy’ is set as a specific objective. It does not define exactly what the terms used mean, nor does it state how the objective should be achieved. Figure 3.5 — Examples of objectives not fit for management purposes DG CLIMA — The amount and deadline are implicitly defined as the objective and measuring indicators clearly relate to the (climate action sub-programme) of the LIFE programme 2014-2020. DG ENV — The Commission notes that the way of how this objective will be achieved is set out on pages 9 and 10 of the Strategic Plan for 2016-2020 of the Directorate-General for Environment and the indicators to measure progress are set out on pages 19 to 21 of the same document.

3.63. In total we reviewed 19 objectives of which only four fulfilled all the ‘SMART’ criteria. This result is similar to what we have found in previous years. While the objectives examined were relevant to the policy area in which the DGs operate, in many cases the indicators accompanying the objective do not exhaustively measure all aspects at the appropriate level.

3.64. The choice of indicators should be based on considerations such as relevance, measurability, and the availability of timely and reliable performance data. As far as is possible, indicators should meet the ‘RACER’ criteria (51). Indicators which measure outcomes over which DGs have limited influence should be complemented by other indicators measuring directly the DGs’ activities.

3.65. All four DGs have at least one performance indicator per objective to help management monitor, evaluate and report on achievements. Of the 58 indicators in the DGs we examined, 28 fulfilled all the RACER criteria. This is an improvement compared to what we have reported in previous years. However, as in previous years, we have identified some shortcomings in the indicators being used (see Figure 3.6 ).

Figure 3.6 — Example of problems with indicators Indicators presented as result indicators but were instead input-oriented, for example in DG AGRI ‘Total investment in renewable energy production’ — This indicator measures the activity directly realised by the intervention instead of focusing on the increase in renewable energy resulting from the investment. Figure 3.6 — Example of problems with indicators This indicator takes into account not only the EAFRD input but also the national and private contributions to the operations. Therefore, it provides an indication of the triggering effects of the EU/national contributions on private contributions in operations directly addressing the renewable energy objective. The 2023 quantified objective is set up ex ante by the Member States based on a needs analysis of the programming area and is meant to be achieved through a consistent strategy. In addition, and above all, this indicator is complemented by the result indicator ‘renewable energy produced from supported projects’ to be measured by evaluators in 2017, 2019 and ex post evaluation (2024). Jointly taken, the two indicators will provide a complete picture on results achieved.

THE COURT’S OBSERVATIONS

PART 3 — RESULTS OF THE COURT’S AUDIT ON PERFORMANCE

2015 special reports and other output on performance

3.66. Our special reports examine whether the principles of sound financial management are applied to EU spending. Within the framework of our 2013-2017 strategy (52), each year we identify a number of priorities on which to focus (53). We choose topics for special reports based on a range of criteria, such as the level of income or spending involved, the risks to sound financial management and the degree of stakeholder interest. In 2015, we adopted (54) 25 special reports, as listed in Annex 3.1 .

3.67. Our approach to performance audit naturally evolves to reflect the changing circumstances in which the EU operates. In this annual report, we draw attention to some of the ways that our 2015 special reports take account of these changes: the development of a series of reports examining different aspects of the financial crisis; a series of reports on the chronic problem of youth unemployment; and our response to developments in climate change and the environment.

EU responses to the financial crisis

3.68. The financial and later sovereign debt crisis which hit in 2007-2008 has had far-reaching consequences for Europe, and the Member States of the EU and the Union as a whole have taken a number of steps in response (55).

THE COURT’S OBSERVATIONS THE COMMISSION’S REPLIES

3.69. In 2015, we published three special reports in this area. Our report on financial assistance to countries in difficulties was a comprehensive analysis of the Commission’s management of the financial assistance provided to five Member States: Hungary, Latvia, Romania, Ireland and Portugal (56). Our conclusions were wide-ranging, drawing attention to the fact that, before the crisis, some warning signs went unnoticed by the Commission, with the result that it found itself unprepared when requests for financial assistance started to arrive. The Commission’s performance in then mobilising resources to deal with the crisis was mixed, although the Commission was generally successful in issuing debt on behalf of the EU to meet financing needs. The impact of the Commission’s action was that the Member States mostly met their deficit targets and generally continued with the reforms that had been sparked by the programme conditions. We made nine recommendations, covering all aspects of this issue. 3.69. The magnitude and nature of the 2007-2008 global financial crisis was unprecedented. The crisis showed that the pre-crisis surveillance framework was not fully adequate in identifying the risk in the underlying fiscal positions in times of unprecedented economic crisis. It highlighted the need for a wider approach to surveillance in the banking sector and wider macroeconomy. At the point when countries applied for assistance, behind-the-scenes monitoring of a deteriorating situation had been underway and the Commission was in close contact with the national authorities. The response to the crisis was immediate and comprehensive. Despite acute staff resource constraints and a steep learning curve, the Commission was successful in quickly taking on all its new programme management duties, not only in issuing debt to meet financing needs in all cases. This is acknowledged by the Court, which qualifies this as an achievement. All programmes achieved their core economic objectives. The audit recommendations focused on process, decision-making and record-keeping aspects of the Commission’s programme management and the Commission accepted, or partially accepted, all the recommendations.

3.70. Greece was one of the Member States most associated with the financial crisis. We therefore published a report focusing specifically on the support provided by the Commission to help Greece implement financial reform (57). We concluded that the body set up by the Commission to deliver its support, the ‘Task Force for Greece’, had achieved its mandate overall in delivering assistance in line with the requirements of the Greek economic adjustment programmes. However, there were shortcomings and mixed results in terms of its influence on the progress of reforms. 3.70. The Commission acknowledges that the impact of technical assistance on the overall progress of reforms was mixed. This needs to be seen in the broader political context and taking into account the division of competences between Member States and the European Union Institutions. The implementation of the (structural) reforms was and remains the responsibility of the Greek authorities, whereas the ‘Task Force for Greece’ was available to advise/assist the Greek authorities where requested.

3.71. The role of credit rating agencies assumed greater significance as a result of the crisis, leading to the establishment of the European Securities and Markets Authority (ESMA) in 2011. Our report on the supervision of credit rating authorities by the EU (58) was therefore an important contribution to more effective management in this area. We concluded that, in a short period of time, ESMA had laid down good foundations for carrying out its supervisory role, but that there remained room for improvement in, for example, clarifying methodological criteria and the adequacy of summary statistics disclosed in and the checks carried out relating to the data reported to ESMA’s online repository.

Youth unemployment

3.72. Chronic youth unemployment is a major problem in a number of Member States, and was an issue we identified as one of the priorities for our work in 2015. In response, in 2013 the Council recommended that a ‘Youth Guarantee’ be established by the Member States, with a combination of EU and national funding. Under the Guarantee, young people, within four months of leaving education or losing a job, should find a ‘good quality’ job offer or training to help them find employment in the future. In our report on this topic (59), we found that the Commission had provided timely and appropriate support to Member States in setting up their guarantee schemes, but pointed to three potential risks that might affect the schemes’ success: the adequacy of total funding, how a good quality offer should be defined, and the Commission’s monitoring and reporting on results. 3.72. Concerning the three potential risks that might affect the youth guarantee schemes’ success identified by the Court, the Commission agrees and would like to point out that: — investing in a youth guarantee is crucial for the EU to preserve its future growth potential. Significant EU financial support is already available, but to make the youth guarantee a reality, Member States also need to prioritise youth employment measures in their national budgets, — ‘good-quality’ is a crucial element for implementing the youth guarantee. The Commission will further stimulate the reflection around ‘good quality’ offers and provide further guidance to Member States, — a lengthy consensus building exercise was necessary to define a common monitoring framework. Monitoring activities will feed into the Commission’s assessment of the youth guarantee implementation under future European Semester and its 2016 report on the implementation of the scheme to the European Parliament and the Council through the appropriate channels.

3.73. We returned to this issue in our report later in 2015 on the Commission’s support for youth action teams (60), an initiative which brought national and Commission experts together in the eight Member States with the highest levels of youth unemployment (61). We concluded that the effectiveness of the advisory support provided by the Commission to the Member States in relation to the youth action teams was limited by the level and quality of information available to it on how existing ESF or national measures for young unemployed people work on the ground. Moreover, the Commission’s assessment of the amendments to their programmes proposed by the Member States focused mainly on budgetary aspects rather than on the best use of the ESF funds available. 3.73. The Commission proactively used all the instruments available to it, given the legal framework in place, to achieve the objectives of the YATs initiative and to report on them. In the Commission’s view, the YATs initiative has met the political expectations set out by the European Council in its conclusions of 30 January 2012. The key objective of the joint YATs — an ad hoc, punctual initiative — was to mobilise EU and national instruments, including structural funds, and to discuss policy reforms to be included in youth job plans. The teams were meant to ‘shake the tree’ by raising awareness at the highest political level, creating a sense of urgency and giving a stronger steer while not creating additional administrative or/and legal procedures.

The environment, energy and climate change

3.74. The environment and climate change, together with its implications for EU policies in a number of other areas, such as agriculture and energy, is one of the global challenges we are targeting in our 2013-2017 strategy. So our report on EU-funded waste water treatment plants in the Danube river basin examined whether EU support was helping Member States achieve its waste water policy objectives (62) — a key element of a successful environmental policy. We concluded that while EU financial support had played an important role in bringing forward water collection and treatment, this had not been sufficient to meet the relevant deadlines.

3.75. A later report in 2015 then examined a different aspect of water quality in the Danube basin: progress by the same four Member States in meeting the requirements of the water framework directive (63), whose main objective from 2000 was to achieve good water quality by 2015 (with some exceptions possible). We concluded that the implementation of measures co-funded by the EU had led to little improvement in water quality. Indeed, while the introduction of river basin management plans as a tool for integrated water management had been a strength of the directive, in practice the Member States’ plans suffered from shortcomings in the identification of pollution pressures and in the definition of remedial measures.

3.76. On a different theme, we examined the EU’s emissions trading scheme (ETS) (64). This is an essential pillar of the EU’s policy to combat climate change, and is designed to promote reductions of greenhouse gas emissions in a cost-effective and economically efficient way. We found that management of the scheme by the Commission and the Member States was not adequate in all respects. It was hindered by some issues related to the robustness of the framework for protecting its integrity, such as the absence of EU level oversight of the emissions market, and by significant weaknesses in the implementation of the second phase of the scheme in the period 2008-2012, after the initial preparatory phase. 3.76. The Commission is committed to continue improving the EU ETS; it will address the shortcomings identified by the auditors. A number of recommendations are to be implemented by Member States. Many points noted by the ECA were specific to the period 2008-12 (phase 2) and have already largely been addressed in the current rules for the period 2013-2020 (phase 3). Since the audit report, there has been a new legislative proposal on ETS (65) (implementation of the climate and energy objectives for 2030 following the October 2014 European Council conclusions).

3.77. We also published a report about the security of energy supply and the development of the EU internal energy market (66). We found that the EU’s objective of completing the internal energy market by 2014 had not been reached. Energy infrastructure in Europe is generally not well designed for fully integrated markets and therefore does not currently provide effective security of energy supply. Financial support from the EU budget in the field of energy infrastructure has made only a limited contribution to the internal energy market and security of energy supply. 3.77. The Commission is taking concrete steps to remove the remaining market barriers to the internal energy market, in particular through regional cooperation frameworks involving different Member States and in the context of preparation of its market design initiative. With regard to infrastructure, it is important to note that: (a) energy infrastructure is expected to be financed by the market, i.e. user tariffs approved by independent regulators; financial support from EU budget therefore should be the exception rather than the rule; (b) where financial support has been allocated, security of supply has often been significantly improved. A good example are the Baltic States where support from both EEPR and CEF to key projects has not only increased security of supply in electricity but also helped wholesale prices to converge.

The follow-up of special report recommendations

3.78. Following up our performance audit reports is an essential element in the cycle of accountability and helps to encourage the implementation of audit recommendations by the Commission and the Member States.

3.79. In our 2016 special report on the 2014 follow up exercise (67), we made a number of recommendations, suggesting that the Commission should bring its follow-up practices into line with relevant internal control standards. The Commission accepted these recommendations and agreed to take corrective actions. 3.79-3.80. The Commission is committed to ensuring that recommendations made by the Court in its special reports and accepted by the Commission are systematically implemented and followed-up. The Commission considers that its control arrangements are in line with international control standards and best practices. However, the Commission is committed to address further challenges concerning the discharge follow-up system and procedures.

THE COURT’S OBSERVATIONS

3.80. We assessed the Commission’s follow up of a sample of 90 audit recommendations from 11 special reports published in the period 2011-2012. The sample was selected on the basis of a range of criteria (68). Of the 83 recommendations that could be verified, we concluded that the Commission fully implemented 63 % of our recommendations, 26 % were implemented in most respects, 10 % in some respects while 1 % were not implemented.

3.81. Although the Commission is our principal auditee, we frequently identify matters which can best be addressed by the Member State authorities or in cooperation with the Commission, given that most EU expenditure is under shared-management arrangements with Member States. In our special reports we sometimes also address recommendations to Member States in general. We can also direct recommendations to Member States individually although this is rare (69).

Recommendations are communicated to Member State level although formal follow-up is mixed

3.82. When we make recommendations applicable to Member States in general, there are five ways in which these messages may be transmitted (see Figure 3.7 ). First, each report is sent to the Supreme Audit Institution (the principal national public audit body). The reports are also sent to the Council, for its consideration in the context of the discharge procedure and, on occasion, through presentations to Council working parties. Our annual activity report (which contains details of our special reports) is also sent to the parliament of each Member State, and individual Members of the Court may bring particular reports to the special attention of a national parliament or its relevant committee, in the course of their interaction with the parliament concerned. Our recommendations are also disseminated to Member States through the medium of Commission guidelines and training interventions.

Figure 3.7 — Communication of findings to Member States Source: European Court of Auditors.

THE COURT’S OBSERVATIONS

3.83. However, to date we have had relatively little information on the impact of these communication efforts in the Member States. To remedy this, we conducted a survey of paying agencies, ministries and SAIs at Member State national level on the follow up of recommendations (70). Four Commission DGs also completed a questionnaire.

3.84. This confirmed that communication of recommendations does indeed get through to Member State level with the survey providing evidence of receipt of our messages through one or more of the above channels. In terms of impact, 80 % of respondents acknowledged at least ‘some awareness’ of our reports at Member State level. However, only 13 % responded that there was a co-ordinated follow-up process for all of our recommendations in their Member State. As regards follow-up action taken, 38 % suggested that action only takes place for particular topics, while 49 % indicated that no formal follow-up action takes place. Nevertheless, some Member States provided concrete examples of embedding our recommendations in national policy and practice, mainly in the field of natural resources (see Figure 3.8 ).

Figure 3.8 — Examples of awareness and communication at Member State level

Our report on ‘The cost-effectiveness of EU Rural Development support for non-productive investments in agriculture’ (71) recommended that Member States should put in place procedures to ensure that the costs of the supported non-productive investments do not exceed the costs of similar types of goods, service or works offered by the market. More concretely, Member States should define appropriate benchmarks and/or reference costs against which the costs of non-productive investments are systematically verified as part of their administrative checks. Responding to this recommendation, one Member State has introduced standard prices for the management of planting of hedges.

For disseminating the Court's recommendations, a managing authority in one Member State makes the Court's findings available on its rural development programme implementation committee intranet where all the implementing bodies and the paying agency are represented.

In one Member State, the Ministry of Agriculture notifies the paying agency and, where necessary, other stakeholders by means of a summary report with our key findings and recommendations, with the full report in an annex.

Source: Select survey responses from our survey of SAIs and paying agencies.

THE COURT’S OBSERVATIONS THE COMMISSION’S REPLIES

3.85. The survey exercise shows that there is a reasonable level of awareness of our performance audit recommendations at Member State level. However, there is a wide variation in the level of formal follow-up and only moderate evidence of changes in national policy and practice as a result of our recommendations. It is clear that there is scope for improvement in how we develop and distribute our recommendations for maximum impact at Member State level. In the autumn of 2016, the Court will engage the Contact Committee, Commission services and the Council in the consultation on possible follow-up of recommendations in Member States.

CONCLUSIONS AND RECOMMENDATIONS

Conclusions

3.86. Horizon 2020 is an ambitious, wide-ranging programme, bringing previously separate research and innovation programmes together for the first time, and incorporating a focus on ‘Societal challenges’. There are high-level links between Horizon 2020 and the EU’s main strategy, Europe 2020: two of the three main indicators for Horizon 2020 are also high-level indicators for Europe 2020. These indicators, however, are of limited use in tracking the contribution of Horizon 2020 to Europe 2020 (see paragraphs 3.7 to 3.17). 3.86. While the Commission accepts that it needs to improve further, it considers that its performance framework already tracks the contribution of Horizon 2020 to Europe 2020. As explained under Figure 3.1 and in paragraph 3.13: The EU budget represents only a small part of total research and innovation spending in Europe, and so in itself can make only a minor contribution towards the achievement of the Europe 2020 objectives. Progress towards the Europe 2020 strategy depends on a wide range of actions at EU and national level, which are not only on spending, and for which Member States have a primordial role to play. The contribution of Horizon 2020 to the achievement of the Europe 2020 strategy is reinforced by the interaction with other actions at EU and national level.

3.87. The links between the Commission’s 10 political priorities and Europe 2020/Horizon 2020 need further clarification. The political priorities overlap with, but are different from, the Europe 2020 priorities. It is not clear to which set of priorities Horizon 2020 — in its planning right through to its monitoring and reporting — is now to be aligned. There is a risk that other areas of EU activity are concerned (see paragraphs 3.18 to 3.21). 3.87. The political priorities of the Juncker Commission provide the framework for the Commission’s own contribution to the delivery of the Europe 2020 strategy, which is a shared responsibility of the Union and the Member States. The political priorities guide the Commission’s implementation of financial programmes but this is done in full complementarity with the Europe 2020 strategy and in accordance with the legal framework underpinning Horizon 2020, which has not changed.

3.88. A key success factor for Horizon 2020 is effective synergy and complementarity between national and European research and innovation programmes. Our partners in the supreme audit institutions of Bulgaria and Portugal found that while there were areas of complementarity in their countries, there were also some problems at the national level (see paragraphs 3.22 and 3.23).

3.89. The legal framework of Horizon 2020 introduces several important elements for performance management such as objectives and key performance indicators. This is a welcome development. However, further developments to the framework are still needed; some baselines, milestones and targets still need to be developed, some targets should be reviewed to ensure that they are suitably ambitious; some indicators suffer from lack of available information. The objectives defined by the legislation are themselves naturally at a high-level, but are not translated, through work programmes and calls, into lower level objectives useful for operational performance management purposes. The Commission does not use the Horizon 2020 work programmes and associated calls to increase the focus on performance (see paragraphs 3.27 to 3.38). 3.89. The performance framework was discussed with the legislative authority during the adoption of the Horizon 2020 legislation, and is set out in that legislation. It includes a number of indicators that will be monitored throughout the programme, and interim and ex post evaluations. This clear framework is an advance on previous framework programmes. The results and impacts of work programmes, and of Horizon 2020 more generally, will be assessed using this performance framework. The long-term nature of research, the timescale of the preparation of the legislation, and the limitations in the availability of some data do provide challenges in the completion of the performance framework, and the Commission accepts that continual improvements are necessary.

3.90. For the proposals and grant agreements we examined, objectives were generally SMART when it was required by the Commission. The evaluation process of these proposals, with the exception of ERC that uses excellence as a criterion, had an emphasis on performance. Grant agreements require beneficiaries to report to the Commission information for the aggregation of the Horizon 2020 KPIs. However the use of the wider concept of expected impact rather than expected result increases the risk that information provided for this part is too broad and not directly attributable to the action funded by Horizon 2020, with the result that performance assessment could be difficult (see paragraphs 3.39 to 3.43). 3.90. The use of ‘expected impact’ rather than ‘result’ arises from the fact that the Horizon 2020 legislation predates the better regulation guidelines. The Commission wants to evaluate the expected impact of projects when considering which projects to support. This will not affect the monitoring of the indicators that have been set out in the legislation, and which are the cornerstone of the performance framework.

3.91. Weaknesses in the design of the monitoring and reporting system for Horizon 2020 mean that the Commission is limited in its ability to monitor and report on the performance of the programme: 3.91. The Commission considers that the Horizon 2020 monitoring and reporting system is a great improvement compared to FP7. Clear objectives have been established, and a performance framework, discussed fully with stakeholders and the legislative authority, has been anchored in the legal basis. The Commission is aware of some weaknesses that remain, and is working to remedy them:

(a) The current setup does not enable the Commission to monitor and report separately the spending and the performance of the different components of research and development and innovation within Horizon 2020 (see paragraphs 3.45 to 3.47). (a) the Commission does not consider that research and innovation spending needs to be reported on separately, this is not required by the legislation and runs counter to the efforts made in Horizon 2020 to close the gap between research and innovation;

(b) The contribution of Horizon 2020 to Europe 2020 is well established in the budgetary process. However, thus far the contribution of Horizon 2020 to Europe 2020 has not yet been reported on in a meaningful way (see paragraphs 3.48 to 3.50). (b) the legislation requires the contribution of Horizon 2020 to Europe 2020 to be reported on in the interim evaluation of the programme, and this evaluation will be undertaken in 2017. The nature of the research cycle means that a certain time must elapse before the contribution to Europe 2020 can be reported on in a meaningful way;

(c) The Horizon 2020 monitoring and reporting system has improved compared to FP7 but there remain weaknesses regarding the information gathered for Horizon 2020. There is a risk that difficulties encountered in the ex post evaluation of FP 7 might persist for Horizon 2020 (see paragraphs 3.51 to 3.53). (c) the Commission agrees that the Horizon 2020 monitoring and reporting system has improved compared to FP7. Clear objectives have been established, and a performance framework, discussed fully with stakeholders and the legislative authority, has been anchored in the legal basis. It considers that it has acted to mitigate the difficulties identified in the FP7 evaluation system.

3.92. In assessing performance, some basic concepts are the different types of benefits which can result from public action: inputs, outputs, results and impacts. While the Commission has defined how these terms should be used, it does not always abide by these definitions. A particular problem is the apparently arbitrary use of ‘result’ and ‘impact’ in translating the legislation into individual work programmes (see paragraphs 3.54 to 3.59). 3.92. The adoption of the Horizon 2020 Regulations (December 2013) preceded the better regulation guidelines (May 2015). This has led to some inconsistencies, as noted by the Court. Since the adoption of the better regulation guidance, the better regulation terminology is consistently used in the monitoring and evaluation of the performance of Horizon 2020.

3.93. For the natural resources family of DGs, we noted that many of the objectives used in MPs and AARs were taken directly from policy or legislative documents and were therefore at too high a level for management purposes. We observed progress for the indicators used to measure performance but some shortcomings remain (see paragraphs 3.60 to 3.65). 3.93. Since the Court’s observation in relation to the management plans for 2015, the Commission has introduced changes to the SPP cycle notably to provide a clearer framework for the Commission’s accountability and for its services. With the new approach, the planning documents as from 2016 are meant to be more streamlined and focused on the priorities of the Commission and the competencies of the DGs, helping them to establish priorities and tell a coherent performance story with a clear distinction between attribution (for results and outputs that are a direct consequence of the Commission services’ activities) and contribution (where the Commission services’ activities play a part in bringing about a given result or an impact) for EU high level policy and spending programmes’ results. The strategic plan introduces a new approach shifting the focus from short-term planning to a long-term, strategic perspective helping the DGs to align their specific objectives with the long-term objectives defined at Commission. The operational planned outputs for the year ahead are listed in the Management Plan.

Recommendations

3.94. Annex 3.2 shows the result of our review of progress in addressing recommendations made in previous annual reports. In the 2012 and 2013 annual reports, we presented five recommendations. Of these recommendations, two were implemented in some respects and three were not implemented.

3.95. Our recommendations for 2015 are as follows:

— Recommendation 1: Operational objectives The Commission should translate the Horizon 2020 legislation high-level objectives into operational objectives at work programme level, so that, by assessing the performance of work programmes and calls, they can effectively be used as drivers for performance. The Commission accepts this recommendation. While the overall performance framework is laid down and clear, the Commission notes the points raised by the Court, which will be fully taken into account as part of work underway to improve the work programmes, including its role in performance based reporting.

— Recommendation 2: Further clarification for the different strategies are needed The Commission should further clarify the links between the Europe 2020 strategy (2010-2020), the multi-annual financial framework (2014-2020), the Commission priorities (2015-2019) through, for instance, the strategic planning and reporting process (2016-2020). This would strengthen monitoring and reporting arrangements and enable the Commission to report effectively on the contribution of the EU budget towards Europe 2020 objectives. The Commission accepts this recommendation. The political priorities of the Commission ar