THE COURT’S OBSERVATIONS THE COMMISSION’S REPLIES

INTRODUCTION

3.1. Each year, in this chapter, we analyse a number of aspects relating to performance: the results achieved by the EU budget, which is implemented by the Commission in cooperation with the Member States (1). This year we have looked specifically at: (i) the performance reporting framework applied by the Commission, (ii) significant results from our 2016 special reports on performance, (iii) the Commission’s implementation of our recommendations in a selection of earlier special reports.

PART 1 — PERFORMANCE REPORTING: HOW DOES THE COMMISSION’S APPROACH COMPARE WITH GOOD PRACTICE?

3.2. Most large and complex organisations have procedures for performance measurement and reporting to provide an indication of the success of operations and highlight where improvements should be made. In paragraphs 3.3-3.12, we provide an overview of the performance reporting framework. In paragraphs 3.13-3.51, we compare it with standards and good practices applied elsewhere.

Section 1 — The performance reporting framework

3.3. The Commission has introduced several changes in this area in recent years — notably by heading up the ‘budget focused on results’ (BFOR) initiative (2). Performance reporting is at the heart of the initiative, which aims to find ways of improving the openness, timeliness and efficiency of communication about the results of the EU budget. 3.3. The performance of the EU budget and related reporting have been reinforced in the current Multi-Annual Financial Framework (MFF). Since the launch of the EU Budget Focused on Results initiative, the Commission has significantly enhanced and streamlined its performance reporting. For example, the Commission produced for the first time in 2016 an Integrated Financial Reporting Package providing detailed information on revenue, expenditure, management and performance of the EU budget in line with the best practices in the fields of transparency and accountability.

3.4. Box 3.1 gives an overview of the performance reporting framework. It reflects the distinction made by the Commission between its own performance and that of the EU budget. In this section we describe the main reports shown in the box. 3.4. At corporate level, the performance reporting of the EU budget encompasses two key reports: the programme statements and the Annual Management and Performance Report. The programme statements accompanying the draft budget are the instrument through which the Commission justifies the allocations of financial resources to the budgetary authority and enables it to take performance information into account. The Annual Management and Performance Report provides to the discharge authorities a high level summary of how the EU budget has supported the European Union's political priorities and of the results achieved with the EU budget, and reports on the Commission's management of the budget.

Box 3.1 — The performance reporting framework applied by the Commission Source: ECA, based on Commission information.

THE COURT’S OBSERVATIONS THE COMMISSION’S REPLIES

A. Programme statements

3.5. The Financial Regulation requires the Commission to prepare ‘programme statements’ (3) to justify the funding requested for EU programmes in the draft budget. For each programme, the statements include an update on implementation, together with key financial and performance information (4), using indicators that will apply throughout the seven-year multiannual financial framework (see Box 3.2 ). This approach enables the budgetary authority to take performance into account during the budgetary procedure. 3.5. The box relates to the 2017 programme statements published in June 2016. They include more than 700 indicators used to monitor the performance of the programmes during their life-cycle. Hence, for a certain number of indicators, data was not yet available. In the 2018 programme statements ( 5 ) , data is available in relation to more than 80 % of the indicators, which represents an increase of more than 20 % compared to the previous year. Performance data will continue to increase with the implementation of the programmes co-financed by the EU budget.

Box 3.2 — 2017 Programme statements: objectives and indicators Number of general objectives Number of specific objectives Number of indicators used Indicators removed for lack of data 2016 Payment Budget (billion euro) Number of indicators per billion euro Heading 1a — Competitiveness for growth and jobs 25 90 154 93 17,4 14 Heading 1b — Economic, social and territorial cohesion 3 21 8 76 48,8 2 Heading 2 — Sustainable growth: natural resources 6 28 29 69 55,1 2 Heading 3 — Security and citizenship 13 48 148 28 3,0 59 Heading 4 — Global Europe 14 44 97 5 10,2 10 Special instruments 0 2 2 0 0,4 5 GRAND TOTAL 61 233 438 271 134,9 5 294 objectives 709 indicators Source: ECA, based on ‘Draft general budget of the European Union for the financial year 2017 — Programme statements of operational expenditure’ (COM(2016) 300 — June 2016).

THE COURT’S OBSERVATIONS

B. Sectoral reports

3.6. In some areas of spending, the TFEU, the Financial Regulation or sector-specific rules require the Commission to prepare sectoral reports. Since the aim, structure and content of these reports vary according to the different requirements, their focus on performance aspects also varies. Some sectoral reports provide wide-ranging, quantified performance information. The Commission monitors some sector-specific performance data online and frequently updates information.

C. Strategic plans, management plans and annual activity reports

3.7. Commission directorates-general (DGs) have prepared multiannual (2016-2020) ‘strategic plans’, which set goals giving medium-term direction to operational activities. In ‘management plans’, DGs break down their multiannual strategy and objectives into annual outputs, actions and interventions. Each DG’s annual activity report (AAR) contains results by reference to the objectives set and an assessment of the extent to which operational expenditure has contributed to policy achievements (6). The AARs also include financial and management information, such as the results of checks on legality and regularity.

D. Evaluations

3.8. In its Better Regulation Guidelines (BRGs) (7), the Commission defines an evaluation as an evidence-based judgment of the extent to which an intervention is: — effective and efficient, — relevant given the needs and its objectives, — coherent both internally and with other EU policy interventions, and — achieving EU added value.

3.9. The BRGs commit the Commission to evaluating, in a proportionate way, all EU spending and non-spending activities intended to have a social or economic impact. Each DG is responsible for evaluating its own activities. Evaluations may be fully centralised in a horizontal unit, fully delegated to operational units or, in a hybrid model, decentralised for operational management purposes but supported by a central evaluation unit.

3.10. Until the end of 2015, the Commission maintained an evaluations database (8) with records of nearly 1 600 evaluations carried out between 1999 and 2015, an average of 100 each year (see Box 3.3 ). The total cost of evaluations in 2016 was more than 60 million euro (210 million euro if studies (9) are included). The Commission has since discontinued the public evaluations database and introduced a ‘studies database’. According to the Commission, the main purpose of the latter is to allow evaluation findings to be shared between the different EU institutions, whereas the public platform for sharing all evaluations and study information is the EU Bookshop.

Box 3.3 — Number of evaluations at the Commission between 2000 and 2015 (three-year rolling average line and annual figures) Source: Commission's evaluations database (2000-2015).

THE COURT’S OBSERVATIONS THE COMMISSION’S REPLIES

E. Annual management and performance report (AMPR)

3.11. The AMPR constitutes a response to the requirements in both Article 318 TFEU (for a results-based evaluation of the EU’s finances) and Article 66(9) of the Financial Regulation (for a summary of annual activity reports). By combining two previously separate reports, the Commission aimed to produce a single document for the discharge authority covering both the management of the EU budget and its evaluation of the results obtained. 3.11. With the publication of the 2015 Annual Management and Performance Report, the Commission has continued to streamline and enhance the performance reporting for the EU budget. The 2016 Annual Management and Performance Report represents further progress in this regard.

F. Other reports with performance information

3.12. There are other Commission reports, both regular and occasional, that include information on objectives and performance indicators, as well as project benefits or achievements.

(i) The annual general report on the activities of the European Union (10), published in March, gives details of the previous year’s major events, initiatives and decisions grouped by the Commission’s ten priorities.

(ii) The annual EU budget financial report (11), published in July as part of the Commission’s integrated financial reporting package, deals with budgetary management in the previous year.

(iii) ‘The Juncker Commission two years on’ (12) was a set of presentations published in November 2016 to provide an overview of the progress made on the Commission’s ten priorities in the two years since it took office.

(iv) The President’s annual ‘State of the Union’ (13) speech to the European Parliament in September, for which supporting documents are published.

(v) The ‘EU Results’ web tool (14), an online collection of EU-funded projects, each with a statement of results and, sometimes, numerical data.

Section 2 — Comparison with good performance reporting practices elsewhere

A. Introduction

3.13. To identify good practices in performance reporting we:

(i) consulted relevant international standards, such as IPSASB Recommended Practice Guideline 3 — ‘Reporting Service Performance Information’ (15);

(ii) carried out a survey on performance reporting; in addition to a comprehensive reply from the Commission, we considered 17 replies from EU Member States, other national governments and other international organisations;

(iii) reviewed publicly available performance information for a selection of 14 governments and international organisations (16);

(iv) examined the assessment reports issued by the Multilateral Organisation Performance Assessment Network (MOPAN) (17) between 2011 and 2016; these always included elements of organisations’ performance reporting.

3.14. While there are important differences between the EU and other international organisations and governments, the latter also have to comply with legal requirements and are subject to specific operating constraints. Other international organisations are no less exposed to complications in establishing a link between their actions and ultimate impact/performance. We identified six areas (B to G below) in which the good practices we observed could help the Commission to further develop its performance reporting. 3.14. As part of the EU Budget Focused on Results initiative, the Commission has launched a dialogue on performance frameworks with relevant experts from national administrations and other international organisations, including the OECD, in order to gather lessons learned and identify good practices. Representatives from the European Parliament and the Council have also participated in this dialogue, which has covered a broad range of issues, e.g. performance, evaluation and EU Added Value. The discussions have clarified that a performance framework should develop gradually and that it cannot be based on a ‘one-size-fits-all’ approach. The Commission is continuing this dialogue with experts and stakeholders and will take the results into account when preparing the performance framework for the next MFF.

THE COURT’S OBSERVATIONS

B. Scope for improving the performance framework

(a) The Commission uses two sets of objectives and indicators to measure the performance, respectively, of its services and of spending programmes

Good practices

3.15. In Australia, government departments produce three core documents as part of the performance framework: portfolio budget statements quantifying programme budgets, corporate plans setting out the purposes and activities of public entities, and annual reports with feedback on results. Entities must ensure coherence (a ‘clear read’) between these documents (18). The performance criteria in each document must be reported collectively in a single annual performance statement (placed in annual reports at the end of the financial year).

3.16. In France, the state budget is subdivided into ‘missions’. Each of the 31 missions in the budget comprises several programmes (118 in 2017). Each programme has a strategy, objectives and quantified performance indicators, and the most important indicators are also identified as mission indicators. The use of identical indicators ensures coherence between missions and programmes. The indicators cover both programme performance and the performance of government departments.

3.17. In Canada, all federal organisations receiving budget appropriations must plan and manage their operations and report their performance against a set of 16 government-wide outcome areas. This framework establishes a clear, straightforward relationship between financial allocations, programme activities and national outcomes.

THE COURT’S OBSERVATIONS THE COMMISSION’S REPLIES

Situation at the Commission

3.18. Before the start of the 2014-2020 MFF, the Commission instructed its services to ‘deepen the performance framework’ (19). To achieve this, it set as an internal rule (20) that the new programme objectives and indicators must feature clearly in programme statements, management plans and AARs. This was done for the first time in 2014, enabling the Commission to present a coherent performance narrative: targets for which money was allocated in the budget became part of the planning process in the relevant DG’s management plan, and the results were reported in its AAR. 3.18. The Commission's Internal Audit Service (IAS) work also brings additional added-value and scrutiny to the Commission's and the various Directorates-General's (DGs) performance frameworks and measurement systems.

3.19. Since 2016, however, the Commission has no longer applied this approach. Instead, it has created a new planning document, the five-year strategic plan, in which DGs are expected to set new organisational objectives and indicators that are aligned with the priorities of the Commission but are distinct from the objectives and indicators already existing for programmes. AARs now report on the performance framework set in the strategic and management plans and should refer to the programme statements for the results of programmes. However, out of the six 2016 AARs we reviewed, only one used cross-references to the performance reported in programme statements. This hampers comparability between the different types of performance documents and does not reflect the fact that the Commission is ultimately responsible for implementation of the budget with regard to the principles of sound financial management. 3.19. The ECA observed in its 2015 Annual Report that many of the objectives in the Commission's management plans were not set at the right level of accountability. The Commission has since reviewed its Strategic Planning and Programming cycle. The objective was to provide a clearer framework for the Commission's/Directorates-General's accountability and to make the planning documents more streamlined and centred on the priorities of the Commission and the competencies of the Directorate-General. The objectives and indicators selected for the strategic plans are now tailored to their specific competences and reflect the fact that the responsibilities of Commission departments are broader than budget execution and programme management. For this reason, aligning objectives and indicators in the performance framework of the Commission services with those defined in the legal bases of programmes is no longer required. However, the Commission services are invited to make reference to programme statements when describing their activities in relation to the programmes. This helps to limit duplication of information and ensure consistency of data and will be reinforced in the future. The Commission considers that its responsibility for implementing the budget is clear. Each Director-General signs in his/her Annual Activity Report a declaration of assurance stating that the resources have been used for the intended purpose and in accordance with the principle of sound financial management and that the underlying transactions are legal and regular. At corporate level, the College takes overall political responsibility for the management of the EU budget by adopting each year the Annual Management and Performance Report for the EU budget.

3.20. The Commission uses a different reporting breakdown for spending and performance. AARs report on DGs’ annual payments by type of activity/spending programme, but on performance by the achievement of general and specific objectives with no indication of the corresponding expenditure. As a consequence, it is not possible to assess how much was spent on pursuing the set objectives. The AMPR similarly does not include information on expenditure relating to the performance of programmes broken down by general and specific objectives. 3.20. The Commission considers that the structure of its reporting in the AARs is appropriate to its institutional and organisational context. General and specific objectives cover not only the management of spending programmes but also non-spending activities (such as policymaking or regulatory activities).

(b) The Commission has a large number of objectives and indicators

Good practices

3.21. In 2016 the OECD carried out a performance budgeting survey in OECD countries and at the Commission. The chart in Box 3.4 compares the existence, use and consequences (21) of performance frameworks in the various jurisdictions. The OECD considered the Commission’s performance framework to be the most extensive, which may partly be explained by the volume of legal requirements in the EU. However, the OECD chart indicates that the use and consequences of the framework for decision-making do not reflect this higher level of specification. 3.21. The performance framework of the EU budget has been developed both to fulfil the legal requirements and to respond to the demands from stakeholders and the Commission's own needs. Performance information on the spending programmes is used by the Commission for (a) monitoring programme progress, (b) accountability and transparency, (c) informing decision-making about the management of a programme; (d) justifying or proposing adjustment of the allocation of funds to the budgetary authorities; (e) developing proposals for changes to the legal basis or for future programmes. The Commission notes however that while performance information is taken into account during the annual budgetary procedure, it is not the only factor driving budgetary decisions.

Box 3.4 — 2016 OECD Performance budgeting survey results Source: OECD presentation at the 12th annual meeting of OECD senior budget officials, performance and results network, 25 November 2016.

THE COURT’S OBSERVATIONS

3.22. Some countries have progressively reduced the complexity of their performance framework and the number of objectives and indicators.

(i) In August 2001 the French Parliament passed legislation (22) establishing new rules for preparing and implementing the state budget. Between 2014 and 2017, this involved the Ministry of Economy and Finance reducing the number of objectives and indicators by 20 % and 24 % respectively.

THE COURT’S OBSERVATIONS THE COMMISSION’S REPLIES

(ii) In 2012 and 2013, the Netherlands introduced an ‘accountable budgeting’ reform which set stricter conditions for the government’s use of policy information (performance indicators and texts explaining policy objectives). The number of performance indicators in budget documents was halved.

Situation at the Commission

3.23. The programme statements for the EU’s 2017 draft general budget contain 294 objectives and 709 indicators (see Box 3.2 ). They are particularly highly concentrated under MFF headings 1a, 3 and 4. Through the BFOR initiative, the Commission is currently undertaking a review of its indicators to provide input for the next generation of spending programmes. Box 3.5 illustrates the progress the Commission could be aiming for, though account should also be taken of certain factors specific to the institution. 3.23. More than 700 indicators are included in the legal bases of the 2014-2020 programmes to help monitor their performance during their life-cycle. They are the outcome of the legislative process and therefore reflect the expectations of the co-legislators as regards the type and level of granularity of information required to track performance. Reporting on these indicators in the programme statements enables the budgetary authority to take performance information into account during the budgetary process. Under the EU Budget Focused on Results initiative, the Commission is reviewing the indicators for the spending programmes as established in the legal basis with a view to drawing lessons for the preparation of the next generation of spending programmes. The comparison made by the ECA does not take account of key contextual factors such as the number and nature of interventions, policy areas covered or the mode of implementation, nor does it measure the quality or relevance of the indicators. The Commission also notes that the number of objectives and indicators used in the strategic plans for the Commission services has been reduced significantly with the recent reform. The number of general objectives has been reduced from 84 to 11, of impact indicators from 187 to 37, of specific objectives from 426 to 386, and result indicators from 969 to 825.

Box 3.5 — The EU has more objectives and indicators (per billion euro) than France or the Netherlands Objectives Indicators per billion euro per billion euro France — Net expenses of the State General Budget 1,0 2,0 Netherlands — State Budget 0,6 2,5 EU — Payment appropriations 2,0 5,0 Source: ECA, based on EU and national budget information.

THE COURT’S OBSERVATIONS

C. Performance reporting is not exhaustive

(a) Performance reporting is not well balanced, as it provides limited information about challenges and failures

Good practices

3.24. Other governments and international organisations have taken steps to ensure balanced performance reporting as recommended by international standards (23).

(i) In the USA, annual performance reports should provide a clear picture of performance results, show trends over time, link to the source of the performance information and discuss progress and shortfalls.

(ii) The WHO ‘end of biennium assessment’ for 2014-2015 included an ‘Overview of major achievements and challenges’ for each programme area (24). In the foreword to the WHO’s main performance report for 2014-2015 (25), the Director-General remarked that ‘Failures as well as successes are frankly presented’. In line with this, the text systematically described the challenges together with the results.

THE COURT’S OBSERVATIONS THE COMMISSION’S REPLIES

(iii) The World Bank assessed (26) the adequacy of its monitoring and evaluation to provide input into the ongoing efforts to enhance its effectiveness and promote a culture for developing solutions and evidence-based decision-making. It identified significant shortcomings in the design and use of its systems.

(iv) MOPAN’s 2015-2016 assessment of the Inter-American Development Bank (27) noted a new section in the bank’s Development Effectiveness Overview called ‘learning from failure’.

Situation at the Commission

3.25. While the 2016 AAR instructions (28) do not explicitly mention the principle of balanced reporting, they do state that ‘the executive summary brings together, in a realistic perspective, the main messages as far as policy achievements are concerned and the conclusions on financial management and internal control’. 3.25. The principle of balanced reporting is further explained in the template for the 2016 Annual Activity Report that accompanied the 2016 Annual Activity Reports instructions ( 29 )

3.26. Notwithstanding the above, the AARs we reviewed contained limited information on the performance shortfalls and challenges relating to the DGs’ objectives. It is also difficult to identify these shortfalls and challenges because no space is specifically set aside for them in the AARs. 3.26. The template for the 2016 Annual Activity Report that accompanied the 2016 Annual Activity Reports instructions requests the Directorates-General to be exhaustive in reporting on all objectives, indicators and outputs (including those whose results show important deviations from set planned values) announced in their 2016-2020 Strategic Plans and 2016 Management Plans.

3.27. With the 2015 AMPR, the Commission set out to provide a comprehensive overview of performance based on an EU budget focused on results. However, the AMPR did not provide comprehensive coverage of performance and was overly positive, the only shortfalls to which it refers being implementation delays. The report also: 3.27. The Annual Management and Performance Report is a high level summary of how the EU budget has supported the European Union's political priorities and the key results that have been achieved with the EU budget. It is not designed to be an exhaustive and detailed report on the performance of the EU budget. The report refers to other performance reports where more detailed performance information can be found (evaluations, programme statements, Annual Activity Reports, etc.). The 2015 Annual Management and Performance Report refers to a number of shortfalls and problems to be addressed. Section 1 of the report refers not only to challenges affecting the timely implementation of programmes but also to limitations in the assessment of programme performance (e.g. references to the evaluations on the 2007-2013 Health programme and programmes in budget heading 4 pointing to constraints in assessing the overall performance of these programmes).

— provided limited insight into the results of the Europe 2020 strategy, whereas this was requested by the European Parliament in its 2014 discharge decision, — The Annual Management and Performance Report is not a specific report on Europe 2020 (30). Nevertheless, the report includes a summary account of progress on Europe 2020 headline targets and relevant indicators and covers both the state of implementation of the 2014-2020 programmes and the results from the 2007-2013 programmes structured around the Europe 2020 priorities of smart, sustainable and inclusive growth. The Commission monitors and reports on the Europe 2020 strategy through the European Semester process. Detailed and comprehensive reporting on Member State performance towards their national Europe 2020 strategy targets is included in the Country Reports as part of the European Semester, as well as in the yearly updates on the Europe 2020 indicators published by Eurostat. The programme statements also provide a link between expenditure and the Europe 2020 strategy. They show the estimated contribution of programmes to the headline targets (theoretical link) and to the Europe 2020 priorities.

— did not always clearly explain the influence of external factors on results, — The 2015 Annual Management and Performance Report describes the operating context by referring in the introduction to the major political and economic challenges of the year. It clearly states that the EU budget is only one instrument, used in conjunction with others (i.e. national budgets and other policy and regulatory tools at EU and Member State levels) to deliver policy results and that 80 % of the EU budget is spent with the Member States. The report concludes under its section 1 on performance and results that ‘…There are limits to the degree to which indicators can capture a precise picture of performance…In addition, contextual factors have an important influence on final results, and indicators cannot eliminate or adjust for these factors’ and that ‘performance is a shared responsibility: responsibility for delivering on the Europe 2020 strategy as well as on the objectives of the financial frameworks is shared to a large extent with Member States’.

— was published too late to be reviewed in our annual report (31). — The deadline for adopting the report under the Financial Regulation is 15 June. The Commission relies on financial data from Member States in relation to shared management programmes which only becomes available in the course of March. It is therefore not possible to adopt the report significantly earlier.

(b) Limited information on performance data quality affects transparency

Good practices

3.28. In the Netherlands, the measurement and reporting of indicators is subject to audit. Annual ministerial reports contain a mandatory feedback section on the reliability of performance information.

3.29. In the US, all government agencies publish a ‘Fiscal Year Annual Performance Report and Fiscal Year n+2 Annual Performance Plan’. This document describes how the agencies will ensure the accuracy and reliability of the data used to measure progress towards their performance goals.

3.30. In Australia, the 2017 Report on Government Services supports the use of imperfect data for the sake of more comprehensive reporting (32). To make up for this deficiency, every indicator in the report contains an assessment of the comparability and completeness of the underlying figures.

3.31. Another aspect of data quality is timeliness. Recent MOPAN assessment reports have described interesting trends in real-time reporting at the Vaccine Alliance, UNAIDS and the World Bank. Our survey (33) also showed that close to half of the respondents report on performance more frequently than once a year (see Box 3.6 ).

Box 3.6 — Reporting frequency to monitor ongoing performance Source: ECA, based on survey on performance reporting.

THE COURT’S OBSERVATIONS THE COMMISSION’S REPLIES

Situation at the Commission

3.32. Currently, the Commission’s performance documents do not systematically include an assessment on the quality of the data used. Deficiencies in data quality may be revealed, but there is no obligation to do so. Performance reports identify data sources, but they do not have a specific section disclosing matters of data accuracy and reliability. According to the Better Regulation toolbox (34), external evaluators can be asked to comment on the quality of data and provide recommendations on how to obtain better data, but this is not a rule. 3.32. The Internal Control Framework put in place by each Commission service is designed to provide reasonable assurance with regard to, inter alia, the reliability of financial reporting. Each Directorate-General reports on this aspect under part 2 of its Annual Activity Report, and section 2 of the Annual Management and Performance Report summarises and reports on the assessment by the Directorates-General on the functioning of the internal control systems of the Commission's services. As to evaluations, the Better Regulation Toolbox stipulates that evaluation Staff Working Documents prepared by the Commission services at the end of an evaluation must contain a clear summary of any insufficiencies in the data used to support the conclusions and the robustness of the results. Moreover, the Better Regulation Guidelines stipulate that any limitations to the evidence used and the methodology applied, particularly in terms of their ability to support the conclusions, must be clearly explained in the evaluation reports. This is further strengthened in the Better Regulation Guidelines/Toolbox revision (SWD(2017) 350 final of 7 July 2017).

THE COURT’S OBSERVATIONS

D. Performance reports are rather narrative and could make better use of visual and navigation aids

Good practices

3.33. Good practices in Australia, in France and at the World Bank are illustrated below.

Box 3.7 — Australia: departmental reports must include tools to help readers navigate Source Australian Department of Social Services, Annual Report 2015-2016 (35) (pp. 253 and 258) and Department of Education and Training, Annual Report 2015-2016 (36) (pp. 225 and 238).

Box 3.8 — France: State General Budget results report uses pictures rather than text Source: ECA, based on the Report on the State General Budget 2015. © Direction du Budget, ministère de l'Action et des Comptes publics, France.

Box 3.9 — World Bank: 2016 annual report uses many tools to enhance accessibility Source: ECA, based on the World Bank’s Annual Report 2016. ‘World Bank. 2016. The World Bank Annual Report 2016. Washington, DC. © World Bank. https://openknowledge.worldbank.org/handle/10986/24985 License: CC BY 3.0 IGO.’

THE COURT’S OBSERVATIONS THE COMMISSION’S REPLIES

Situation at the Commission

3.34. The Commission uses quantitative and qualitative data to present performance information through a combination of graphics and narrative descriptions. Box 3.10 shows that — for a selection of reports — although charts, graphs and tables are common, textual information predominates. The Better Regulation toolbox has a section on the use of visual aids, but this is limited to only three types (problem trees, objective trees and intervention logic diagrams). Other, more general tools for enhancing visual presentation — such as colour-coding or infographics — are not discussed. 3.34. The Commission recognises the importance of using visual aids in its various performance reports and has made improvements in this area. For example, the 2015 version of the Annual Management and Performance Report uses textboxes to make key messages and examples more visible and facilitate reading of the report. The 2016 edition of the Annual Management and Performance Report has further improved the presentation using graphics, charts and images and including an Executive Summary. This and other reports in the Integrated Financial Reporting Package are accompanied by one-page fact sheets which provide an easily accessible summary of the key information for the reader. Moreover, the instructions for the Annual Activity Reports invite the Directorates-General to use visual elements in the Annual Activity Reports to make those documents more easily readable. The Better Regulation Toolbox includes a dedicated tool on visual aids. This tool aims at presenting some examples of instruments that are considered particularly relevant for policy interventions as a way to raise awareness among Commission services of the added value of such aids. It is by no means meant to bring together all existing visual aids.

3.35. Navigation aids — for example glossaries, lists of abbreviations and alphabetical indexes — are almost never used. The Commission has not adopted the practice of producing interactive online versions of key performance reports.

Box 3.10 — Use of visual and navigation aids in the Commission’s performance reports Number of pages Number of graphs and charts Number of tables Colour coding Interactive online version Visual summary of core messages Glossary Alphabetical index 2015 AMPR 58 8 2 No No No No No DG ENER 2015 AAR (without annexes) 99 27 19 No No No No No DG HOME 2015 AAR (without annexes) 64 6 18 No No No No No DG MOVE 2015 AAR (without annexes) 123 13 23 No No No No No DG TRADE 2015 AAR (without annexes) 38 3 5 No No No No No The EU in 2016 95 18 0 Yes No Yes No No Source: ECA, based on Commission reports.

THE COURT’S OBSERVATIONS THE COMMISSION’S REPLIES

E. The Commission does not demonstrate that it systematically uses evaluation results

(a) Recommendations not always made

Good practices

3.36. The UNESCO evaluations we reviewed put forward recommendations together with ‘strategic options for consideration’. All recommendations follow the same pattern: they define a goal and the means for achieving it, followed by a list of actionable suggestions for implementation.

3.37. A recent WHO evaluation (37) assesses the reach, usefulness and use of a sample of approximately 15 000 publications over 10 years. The evaluation makes six strategic recommendations and presents specific actions side-by-side with the text of each recommendation to explain what they entail and promote follow-up.

Situation at the Commission

3.38. The BRGs do not require evaluators to make recommendations for consideration by the Commission; we consider that this should be the basis for a well-functioning follow-up system including action plans addressing weaknesses. The practice on making recommendations varied among the twelve evaluations we reviewed: — six evaluations contained recommendations and proposed implementing actions; — four evaluations contained general recommendations but proposed no concrete actions to improve the situation; — two evaluations contained no recommendations. 3.38. The Better Regulation Guidelines provide that all evaluation Staff Working Documents include findings and conclusions which are the basis for possible follow-up action by the Commission. They also include requirements for the dissemination of evaluation findings and the identification of appropriate follow-up actions. A formal report to the legislator typically sets out follow-up actions under consideration by the Commission.

3.39. The Commission has not carried out a study on its use of evaluation results, or had one made, since 2005. 3.39. In the 2015 Better Regulation package the Commission committed to assess the functioning of the system before the end of 2019. Preparatory work for this assessment is now starting.

(b) The Commission does not demonstrate that evaluation recommendations are systematically followed up

Good practices

3.40. We found several good practices in the follow-up of evaluations and how it is reported:

(i) The World Bank tracks and reports progress on the implementation of evaluation recommendations, including management responses and action plans with indicators, targets, deadlines and responsible entities, through the Management and Action Record (38). This allows aggregate conclusions to be drawn concerning the typology and implementation of recommendations (39).

(ii) United Nations agencies react to evaluation recommendations by providing management responses and timed action plans identifying responsible units.

(iii) Canada’s policy on results (40) requires evaluation reports to contain management responses and action plans. The three evaluations we reviewed did this, and also designated responsible bodies, set a completion date and defined deliverables.

Situation at the Commission

3.41. The Commission has no documented institutional system for the regular follow-up of evaluations. Since 2015, the BRGs (41) have required a staff working document summarising and presenting the final results of the evaluation process to be prepared for all evaluations. The Commission has not yet fully implemented this new requirement. 3.41. Follow-up to evaluations is an intrinsic part of the Better Regulation system in the Commission whereby evaluation results feed into impact assessments, which are publicly available documents, and the annual programming and planning of Commission activities. The Commission follows up on the results of its evaluations in the framework of the Commission Work Programme. In planning the work programme, the Commission systematically reviews evaluation results and determines their follow-up. The follow-up to some evaluation findings are also set out in the REFIT scoreboard which is updated annually and in Commission reports to the legislator accompanying evaluations.

3.42. The BRGs specify that management plans should record the action taken to follow up evaluations, whatever form these may take (42). However, this is not a requirement in the relevant instructions, and in practice the 2016 management plans established no basis for monitoring such action. As the Commission does not have an overview of the conclusions, recommendations or action plans resulting from its evaluations, or track their implementation at institutional or DG level, it cannot inform stakeholders about the positive impact of evaluations. 3.42. The instructions for the 2018 Management Plans will include explicit reference to the relevant section of the Better Regulation Guidelines. The results of some evaluations and their follow-up in the legislative cycle are presented in the REFIT scoreboard. The scope of REFIT has been extended considerably in 2017. Stakeholders are invited to provide feedback on the results of evaluations and on follow-up through the Better Regulation Portal. It is foreseen to fully integrate the REFIT scoreboard into this Portal in future.

F. Core performance reports do not include a declaration or information on the quality of performance information

Good practices

3.43. Performance statements are given in different forms and at different levels of responsibility. In terms of scope and/or level of responsibility, the practice of some governments goes beyond the declaration(s) given at the Commission:

(i) Section 39 of Australia’s Public Governance, Performance and Accountability Act of 2013 (43) stipulates that ‘The accountable authority of a Commonwealth entity must prepare annual performance statements…’.

(ii) In the USA, performance declarations vary. The Department of Education 2015 Annual Performance Report and 2017 Annual Performance Plan include the following declaration by the Secretary of Education: ‘To the best of my knowledge, the data verification and validation process and the data sources used provide, to the extent possible, complete and reliable performance data pertaining to goals and objectives in our FY 2014-18 Strategic Plan. …’.

(iii) In the UK, the Accountability report which is part of the Annual Report and Accounts presented by departments includes a statement of the accounting officer’s responsibilities. The relevant paragraph of this statement reads as follows: ‘I also confirm that this annual report as a whole is fair, balanced and understandable, and I take personal responsibility for the annual report and accounts and the judgments required for determining that it is fair, balanced and understandable’.

Situation at the Commission

3.44. In each AAR, the Director-General declares that the information in the report gives a true and fair view (44). However, it is not made clear in the declaration that this assurance does not extend to the information on the results achieved (45). 3.44 and 3.45. The overarching principles of the chain of accountability are enshrined in the EU treaties. Within the Commission, following the White Paper on governance, the accountability chain has been defined with a prominent role for the Authorising Officers by Delegation within a decentralised system operating under the political responsibility of the College as Authorising Officer. This is particularly true for the organisation of financial management. At the end of the reporting cycle, the College adopts the Annual Management and Performance Report for the EU Budget and takes overall political responsibility for the management of the EU Budget based on the annual declarations of the Authorising Officers by Delegation. The Commission statement about the responsibility for the management of the EU budget is included to make it clear that the Commission has the ultimate financial responsibility for the management of the EU budget, whereas the responsibility for the results achieved with the EU budget is shared with a wide range of actors at European and national level. See also Commission reply to paragraph 3.32.

3.45. Because AARs do not include a declaration on the quality of the reported performance data, a similar approach is taken in the AMPR. In adopting the latter, the College of Commissioners takes overall political responsibility for the management of the EU budget (46) but not for the information on ‘Performance and results’ (47).

G. Performance information provided by the Commission is not easily accessible

Good practices

3.46. In the Netherlands, the Ministry of Finance has a single website (48) that carries all official Dutch budget documents with information on the planning and achievement of results, lessons from evaluations and major policy changes. Documents are sorted by type and year (and other relevant sub-categories) to help navigation. Brief descriptions give an overview of the purpose and content of the various document types.

3.47. The US Government operates a website (www.performance.gov) to inform the public about the progress of major federal agencies towards their shared and individual objectives. Results are presented by agency, goals, management initiatives and programmes.

3.48. The UK Government tracks the performance of government services through a ‘performance platform’ (https://www.gov.uk/performance). The platform is part of the central government website, which also includes a database of government publications (49). The database has a user-friendly search interface and several useful filters (publication type, policy area, department, official document status, world location, and publication date).

3.49. The World Bank has a central hub (50) for locating information on the performance of the Bank and the development areas in which it operates. To make results more accessible, they are grouped into simple categories and often presented as links to graphs on other websites. Accessibility is also enhanced by the fact that readers can choose the desired level of detail — see for example the site of the President’s Delivery Unit (51).

Situation at the Commission

3.50. There is no central performance website with information from all Commission departments on every area of the EU budget. The ‘Publications’ website (52) serves as an access portal for various information sources, but it lacks a map to explain what is available where, for what purpose, and how the different elements fit together. The website is also incomplete: the list of strategic documents does not include programme statements (see paragraph 3.5) or sectoral and other performance reports. 3.50. Under the EU Budget Focused on Results initiative, a specific EU Results website has been developed. The website is a database which aims to become the single entry point to all EU funded projects — via direct, indirect and shared management. By the end of May 2017, the database has hosted about 1 600 EU funded projects in the EU and outside.

3.51. Among its 110 000 titles, the EU Bookshop includes Commission studies and evaluations. The bookshop has no dedicated section for these documents (in the same way as for ‘authors’, ‘themes’ or ‘e-books’), so the only way to find a study or an evaluation is through an advanced search function. Although the search function has filters for studies or evaluations, hits are not presented in a user-friendly manner (e.g. documents supplementing studies or evaluations are shown as separate hits). 3.51. The Commission is continuously working to improve the presentation of studies and evaluations on the EU Bookshop webpage with the aim of making the content clearer and more easily accessible for the external reader. The current version of the EU Bookshop will soon be phased out to be replaced by a new improved platform.

PART 2 — RESULTS OF THE COURT’S PERFORMANCE AUDITS: COMMON CHALLENGES IDENTIFIED IN SOME 2016 SPECIAL REPORTS

Introduction

3.52. Each year we produce a number of special reports in which we examine how well the principles of sound financial management are applied when implementing the EU budget. In 2016 we adopted 36 special reports (53) — more than in any other year so far (see Box 3.11 ). They covered all MFF headings (54) ( Box 3.12 ) and contained a total of 337 recommendations ( Box 3.13 ). Our 2016 special reports also included positive observations specific to each individual area examined. The published replies to our reports show that nearly three quarters of our recommendations were fully accepted by the auditee, which was usually the Commission ( Box 3.14 ). 3.52. The Commission notes that the ECA's overview of common challenges is based on the situation presented in the special reports at the time of their presentation and does not take into account action taken by the Commission since then. Some of the findings highlighted by the ECA in paragraphs 3.54-3.59 have therefore already been addressed.

Box 3.11 — More special reports than ever before (three-year rolling average line and annual figures) Source: ECA.

Box 3.12 — Our special reports cover all MFF headings and more Source: ECA.

Box 3.13 — Recommendations cover a wide range of topics Source: ECA.

Box 3.14 — How persuasive are we? Source: ECA.

THE COURT’S OBSERVATIONS THE COMMISSION’S REPLIES

3.53. The priorities established by the Court in its 2013-2017 strategy for focusing its work are: (i) sustainability of public finances, (ii) the environment and climate change and (iii) added value and growth. In the following paragraphs we draw attention to a selection of special reports that relate to these priorities. The chosen reports represent about one third of the 36 reports for 2016. From these reports, we present a selection of common challenges that led to recommendations, without attempting to reiterate the overall conclusions. Boxes 3.15 , 3.16 and 3.17 give an overview highlighting these challenges for each priority of the 2013-2017 strategy. 3.53. The Commission underlines that for some of the special reports selected in Boxes 3.15, 3.16 and 3.17, the ECA has also identified good results and achievements for example: — In SR 19/2016, the ECA identified that with regard to management costs and fees, the legislation was significantly improved, providing ceilings on cumulative amounts which are below those applicable during the 2007-2013 programme period. — In SR 10/2016, the ECA identified that in the recent years, the Commission has made commendable efforts to improve transparency. — In the SR 31/2016, the ECA identified that ambitious work was underway and that, overall, progress had been made towards reaching the target. The implementation of the target has led to more, and better-focused, climate action funding in the European Regional Development Fund and the Cohesion Fund. — In SR 4/2016, the ECA identified that thanks to the European Institute of Innovation and Technology (EIT), the three Knowledge and Innovation Communities (KICs) launched in 2010 have brought more than 500 partners together across disciplines, countries and sectors. KICs are autonomous, an EIT feature particularly valued by the KIC partners. Through the EIT, the KICs have offered new opportunities to carry out innovative activities and have promoted an entrepreneurial culture. The ECA welcomed the EIT’s decision to review its indicators in 2015 by implementing a new key performance indicator management system.

Sustainability of public finances

3.54. Three special reports adopted in 2016 dealt with the sustainability of public finances. Although focusing on different instruments designed to safeguard the EU’s or Member States’ budgets, they identified some common implementation weaknesses. Two elements of our findings (concerns over the quality of data and over transparency) are issues we identified in many subject areas. 3.54. The Commission highlights the following in relation to ‘guidance’ and ‘data quality’ mentioned for the SR 19/2016 in Box 3.15: At the moment of publication, the Commission was providing extensive guidance, manuals and possibilities for exchange of experience. These are all available under https://www.fi-compass.eu/resources/ec. Topics covered include inter alia ex ante assessment, payment modalities, management costs and fees, combination of ESIF and EFSI, selection of bodies implementing financial instruments. The topics were selected based on the need for guidance expressed by the Member States. The cases mentioned in the ECA report, except the guidance on preferential treatment (presented to Member States in October 2015), are linked to recommendations for the Commission to provide additional guidance to Member States to make use of economies of scale, respect the tax provisions or to take appropriate measures to maintain the revolving nature of the funds during the required eight-year period after end of the eligibility period for the 2014-2020 programming period. The only reference in SR 19/2016 with regard to the quality of data is in relation to the management costs and fees reported by the managing authorities. For a number of financial instruments, the ECA report indicates that data on management costs and fees was either not reported at all or was not plausible. A mandatory reporting on the management costs and fees incurred and paid was legally foreseen only at closure, by March 2017. The Commission also underlines that, building on the experience of the 2007-2013 period, requirements for the 2014-2020 period are much more detailed and a more structured provision of data is foreseen.

Box 3.15 — Common problems identified in reports on the sustainability of public finances Guidance Data quality Transparency SR 10/2016 — Excessive deficit procedure (EDP) X X X SR 19/2016 — Implementing the EU budget through financial instruments X X N/A SR 29/2016 — Single Supervisory Mechanism (SSM) X X X Source: ECA.

THE COURT’S OBSERVATIONS THE COMMISSION’S REPLIES

3.55. Examples of common findings in these reports 3.55.

(i) Insufficiently developed guidance (SR 29/2016): to improve the decision-making process of the SSM Supervisory Board and ensure the efficiency and effectiveness of Board meetings, further guidance should be developed in the form of checklists, templates and flowcharts. (i) Insufficiently developed guidance (SR 29/2016): The Commission notes that SR 29/2016 did not contain recommendations addressed to the Commission but to the ECB.

(ii) Data quality (SR 19/2016): the Commission did not have a comprehensive overview of the management costs and fees for implementing European Regional Development Fund and European Social Fund financial instruments. Similarly, the Commission lacked information on the additional overall administrative costs for each Member State of implementing the European Structural and Investment Funds through grants or financial instruments during 2014-2020. (ii) Data quality (SR 19/2016): For 2007-2013, the Commission monitors the management costs and fees through the summary of data for financial instruments based on the reporting from Member States. This requirement was introduced in 2011. For 2014-2020, the Commission will have a more complete and reliable set of data to monitor the whole financial instruments’ implementation, including management costs and fees. For both periods, the national authorities are responsible for ensuring the eligibility of the management costs and fees in line with the applicable rules and the principle of subsidiarity.

(iii) Transparency (SR 10/2016): in general, the analyses underlying the Commission’s proposals for EDP decisions and recommendations lacked transparency. We identified a need to balance the increased complexity and wider scope for economic judgement by enhancing transparency and thus facilitating public scrutiny. Eurostat did not report to the Economic and Financial Committee or make public all its ex ante/ex post bilateral advice to Member States. The reasons for Eurostat reservations and amendments, and the procedures underlying those reservations, could also be made more transparent. (iii) Transparency (SR 10/2016): Eurostat has a procedure specifying the steps, timetable and role of each actor for the analysis of the notified EDP data. This includes regular meetings with senior management and with the Director-General. As an outcome of the ECA audit, Eurostat does in fact, since summer 2016, publish ex ante and ex post advice given to Member States on its web site, and they are reported to the EFC. They are available in the following section of the Eurostat's web site: http://ec.europa.eu/eurostat/web/government-finance-statistics/methodology/advice-to-member-states Concerning the reasons for Eurostat reservations and amendments, which are based on professional judgement, Eurostat has established procedures for setting reservations and amendments to the data, as an outcome of the ECA audit.

Environment, energy and climate change

3.56. In 2016 we published four reports on environment, energy and climate change. While these reports addressed different topics, they also raised some common issues. As in other areas, data quality and monitoring emerged as common areas of concern. Concerns over timeliness also emerged in this area. 3.56. See Commission reply to paragraph 3.57.

Box 3.16 — Common problems identified in reports on the environment, energy and climate change Data quality Timely implementation Monitoring and oversight SR 3/2016 — Nutrient pollution in the Baltic X X X SR 18/2016 — Certification of biofuels X N/A X SR 22/2016 — Nuclear decommissioning in Lithuania, Bulgaria and Slovakia N/A X X SR 31/2016 — EU climate action X X X Source: ECA.

THE COURT’S OBSERVATIONS THE COMMISSION’S REPLIES

3.57. Examples of common findings in these reports 3.57.

(i) Data quality (SR 3/2016): the procedure for monitoring pollution from nutrient inputs in the Baltic Sea did not guarantee the reliability of data for the purposes of assessing compliance with HELCOM (55) targets.

(ii) Timeliness (SR 22/2016): nearly all key infrastructure projects in the three audited Member States had experienced delays. The longest delays were in Lithuania, where the decommissioning end-date had been postponed by nine years to 2038; (SR 31/2016): we found that ambitious work was underway to reach the target to spend at least 20 % of the EU budget for 2014-2020 on climate-related action, but there is a serious risk that this target will not be met. Overall, the Commission estimates that 18,9 % would be spent on climate action, thereby falling short of the 20 % objective. (ii) Timeliness (SR 22/2016): The Commission noted that the decommissioning programmes are at different level of advancement and maturity in the three Member States. The Bohunice (SK) and Kozloduy (BG) programmes are the most advanced and scheduled for completion in 2025 and 2030 respectively. The latter programme was shortened by 5 years when it was revised in 2011. In Ignalina (LT), the decommissioning of the Chernobyl-type reactors is a first-of-a-kind process which actually entails the greatest challenges. Delays were incurred in the past multi-annual financial framework. Controls and management structures have subsequently been reinforced to mitigate the issues encountered. Notwithstanding the progress already achieved, the Commission recognised the need for continuous improvement in the decommissioning programmes.

(iii) Monitoring (SR 18/2016): the Commission was not supervising the operations of voluntary biofuel certification schemes and therefore could not be sure that these actually applied the standards for which they had been certified or detect infringements of the rules. (iii) Monitoring (SR 18/2016): The Commission noted that its ‘supervisory power’ regarding the implementation of all the operations was limited by the legislator. The Commission is supervising the voluntary schemes in line with the legal competences provided by the Renewable Energy Directive as amended by Directive (EU) 2015/1513 which requires voluntary schemes to report annually on their operation. The Commission agreed that supervision and transparency, including complaints procedures, could be strengthened.

Added value/cost reduction

3.58. The creation of added value is a recurrent theme in all our special reports, and we were led to make recommendations in many cases about the development of legislation or procedures or, more rarely, the reduction of costs. We identified common problems in this respect in four reports on EU institutions, bodies and agencies. Concerns over monitoring and transparency were also frequent in this subject area, as in many of our reports.

Box 3.17 — Common problems identified in reports on EU institutions, bodies and agencies Added value/cost reduction Monitoring Transparency SR 4/2016 — European Institute of Technology (EIT) X X X SR 7/2016 — European External Action Service (EEAS) buildings X X N/A SR 12/2016 — Agencies’ use of grants X X X SR 17/2016 — EU institutions’ procurement N/A X X Source: ECA.

THE COURT’S OBSERVATIONS THE COMMISSION’S REPLIES

3.59. Examples of common findings in these reports 3.59.

(i) Added value (SR 4/2016): despite a valid raison d’être, the EIT’s complex operational framework and management problems were impeding its overall effectiveness and it should refocus its approach to delivering impact. (i) Added value (SR 4/2016): The EIT has adopted a series of measures to address shortcomings in effectiveness. The EIT has addressed the complexities in the operational framework through the Task Force for Simplification, the EIT monitoring strategy and the good governance and financial sustainability principles. Finally, the EIT has strengthened its leadership, management and governance structures.

(ii) Monitoring (SR 7/2016): the EEAS did not take all relevant factors into account when monitoring the available space per person in its buildings, nor did it always ensure that the rents it paid for office and residential space were in line with market rates, or that charges passed on to other tenants recovered costs in full.

(iii) Transparency (SR 12/2016): we found shortcomings where the audited agencies applied specific expert selection and grant award procedures based on exemptions in the founding regulation; as a result there were risks to the principles of equal treatment and transparency, and potential conflict of interest issues had not been fully addressed; (SR 17/2016): the Internet visibility of the EU institutions’ procurement activities was poor; information on the results of procurement was not accessible to effective monitoring by the discharge authority and the wider public to increase transparency and build confidence.

THE COURT’S OBSERVATIONS

PART 3 — FOLLOW-UP OF RECOMMENDATIONS

3.60. Our follow-up of the recommendations in our audit reports is an essential step in the performance audit cycle. As well as providing us and other stakeholders (chiefly the European Parliament and the Council) with feedback on the impact of our work, follow-up helps to encourage the Commission and Member States to implement our recommendations.

3.61. This section contains the results of our yearly review of the extent to which the Commission has taken corrective action related to our recommendations.

Scope and approach

3.62. This year we selected 13 reports that:

— we published at least three years ago,

— were still relevant, and

— we have not previously followed up in a special or an annual report. The reports contained a total of 131 recommendations issued between 2010 and 2013 on a variety of topics (see Box 3.18 ). Details on their implementation status are given in Annex 3.1 .

Box 3.18 — More recommendations followed up than in any previous year Source: ECA.

THE COURT’S OBSERVATIONS

3.63. We carried out our follow-up on the basis of documentary review and interviews with Commission staff. To ensure a fair and balanced review, we then sent our findings to the Commission and took account of its replies in our final analysis. We concluded on the implementation of 108 recommendations. We could not conclude on the remaining 23 either because they did not directly address the Commission or because they were no longer relevant.

How has the Commission addressed our recommendations?

3.64. Of the 108 recommendations, we concluded that the Commission had fully implemented 72, while 18 were implemented in most respects, 12 in some respects, and six were not implemented (see Box 3.19 ).

Box 3.19 — High number of recommendations implemented Source: ECA.

THE COURT’S OBSERVATIONS

3.65. This outcome is broadly in line with previous years. However, the percentage of fully implemented recommendations was the highest since we started to publish consolidated figures.

3.66. Even where not fully implemented, our recommendations had frequently triggered corrective measures (see Box 3.20 ). A large majority of these were in place within two years.

Box 3.20 — Recommendations frequently led to significant corrective action Source: ECA.

3.67. We were unable to verify 21 recommendations (16 % of the overall total) because they were addressed solely to Member States (56).

3.68. The Commission stated that it would review significant issues with the Member States for specific cases in shared management where it had a supervisory role to play. A Court/Commission joint working group is currently discussing a possible mechanism for reporting and verifying these results. We also plan to work with the Contact Committee (57) and the Council to identify collaborative arrangements which might yield further useful information on Member States’ implementation of recommendations.

THE COURT’S OBSERVATIONS THE COMMISSION’S REPLIES

A mixed bag of results

3.69. Our follow-up shows that our recommendations led to several key improvements. Unresolved weaknesses are described in Annex 3.2 .

3.70. Six recommendations were not implemented (see report titles in Annex 3.1 ): 3.70.

— SR 20/2012 (paragraph 77(a)). The 2015 legislative proposals on waste did not take up our recommendation to set binding targets at EU level. This is because the Commission is moving from setting targets to supporting actions for the prevention of waste, such as food waste reduction and eco-design. — SR 20/2012. The Commission continues to legally and financially support waste prevention as the highest rank of the waste hierarchy through, for example, EU Cohesion funds. Whether or not the revised waste legislation will contain references to waste prevention (including reuse) targets cannot be predicted at this stage as the legislative process on the Commission's waste proposals is still ongoing (and will continue during the second half of 2017).

— SR 14/2012 (paragraph 54(b)). No evaluation of the impact of the EU funds allocated to the implementation of hygiene and food safety in slaughterhouses has yet been carried out. — SR 14/2012. If Member States have covered the food safety and hygiene standards in their ex post evaluation of the 2007-2013 Rural Development Programmes, the Commission will include the results of it in its summary evaluation. In any case, the measure has been discontinued in the current programming period.

— SR 14/2012 (paragraph 54(c)). Given that the evaluation referred to above has not been made, it was not possible to consider whether further action should be taken. — SR 14/2012. In the current programming period (2014-2020), the measure has been discontinued.

— SR 2/2013 (first part of recommendation 7, paragraph 104). The Commission had not implemented effective measures to guarantee the additionality of the successor to the Risk Sharing Finance Facility. This was confirmed in November 2016 by an independent evaluation which expressed concerns about the non-additionality of 28 % of projects and recommended establishing clear selection criteria to guarantee additionality. — SR 2/2013. It will always be difficult to ensure the additionality of every project ex ante, this assessment depends on a range of factors that are continually changing, and linked to changing economic conditions. However, 2014-2020 financial instruments were improved in terms of evaluation, monitoring and reporting requirements and the interim evaluation of INNOVFIN has been designed to provide answers to the concerns of the ECA. Difficult access to finance remains a concern for SMEs.

— SR 12/2012 (paragraph 108(d)). Contrary to our recommendation, the Director-General of Eurostat is still appointed without the prior involvement of the European Statistical Governance Advisory Board or the European Parliament and the Council. — SR 12/2012. An appropriate legal framework and necessary safeguards exist to ensure that appointment and dismissal procedures concerning Eurostat's Director-General are transparent, ensuring full compliance with the principle of independence as foreseen in Regulation (EC) No 223/2009 in this respect. The transparency has been confirmed by the recent procedure concerning the post of Director-General of Eurostat which was open to external applicants. In addition, ESGAB's independent reporting on the implementation of the Code of Practice by the Commission (Eurostat) is best served by it not being directly involved in the appointment of Eurostat's Director-General. Finally, the inter-institutional relations between the European Parliament, the Council and the Commission are well established, for instance as regards legislative procedure but also by making the Commission as a whole accountable before the Parliament. The annual statistical dialogue foreseen in Regulation (EC) No 223/2009 aims at ensuring appropriate involvement of, and information to, the Parliament on statistical matters, including by foreseeing that the newly appointed Director-General of Eurostat shall appear before the relevant committee of the Parliament immediately after appointment.

— SR 23/2012 (paragraph 73(e)), last sentence). The Commission did not initially accept this recommendation, on following up the use of reimbursement clauses in grant decisions for regeneration projects, on the grounds that follow-up was the responsibility of Member States’ managing authorities. — SR 23/2012. As mentioned in its reply to this recommendation, the Commission considers that the inclusion of a reimbursement clause in the grant letter issued by the managing authority is a case of good practice which can usefully be included by Member States in their national rules. However, under the principle of shared management, since the Managing Authorities issue the grant decisions, they should also monitor the application of the reimbursement clause as they monitor the implementation of projects.

3.71. The Commission had initially not accepted 11 recommendations, nine of which concerned DG DEVCO (including six from SR 9/2013). Nevertheless, all but one were subsequently implemented partially or in full (see Box 3.21 ). 3.71. After the 2015 reorganisation of the Commission, two of the nine initially rejected recommendations by the Directorate-General for International Cooperation and Development (DG DEVCO) fall under the remit of Directorate-General for Neighbourhood and Enlargement Negotiations (DG NEAR). The rest of the seven initially rejected recommendations were covered by the 2016 follow-up exercise of the ECA: — one recommendation from the SR 13/2013, requiring the Commission and the European External Action Service (EEAS) to set up a new system for calculating and reporting on the overall administrative costs involved in delivering development assistance aid for the Central Asia was rejected being considered that a change to the existing ABB system should have been raised by the ECA at the Commission level and further agreed with the budgetary authority; — in the SR 9/2013, DG DEVCO and the EEAS have not agreed with 6 out of 12 recommendations, due to the fact that most of the actions were considered as being already implemented as explained in the EEAS and Commission's joint responses to the ECA. Therefore this positive outcome should not be surprising. See Commission reply to paragraph 3.70 in relation to SR 12/2012.

Box 3.21 — Almost all recommendations not initially accepted were nevertheless implemented SR number Rejected and not implemented Rejected and partially implemented Rejected and fully implemented SR 23/2012 Regeneration The polluter-pays principle is now a condition for the funding of regeneration projects. SR 14/2013 Palestine Authority The Pegase programme of direct financial support to Palestine was initially provided without explicit conditionality; this changed with the introduction of a results-oriented approach. SR 13/2013 Central Asia A new system for calculating and reporting on the overall administrative costs of delivering development assistance in Central Asia was not felt to be necessary because the existing activity-based budgeting system was sufficient. SR 9/2013 Democratic Republic of Congo A standard timeframe was not agreed for governance projects in the Democratic Republic of the Congo because the Commission preferred to proceed on a best-knowledge basis and projects could be amended or extended. However, realistic time-bound indicators have generally been put in place. Although our recommendation concerning the balance of aid in all provinces of the DRC was not accepted, we found a reasonably fair balance of aid between central and outlying regions. Rather than strengthen its use of conditionality and policy dialogue, the Commission said that it applied the principles agreed internationally for fragile states. We found nonetheless that conditionality and policy dialogue are in place, although they are not yet fully effective. Measures to prevent and mitigate risks did not need to be established because the Commission and the EEAS would tailor these to the developing situation. We found that for the new programming period the Commission has taken greater account of the possible risks and intensified its mitigating measures accordingly. The Commission did not accept the need for further flexibility during programme implementation since programme amendments were part of normal practice. We found that programmes are now adjusted on the basis of regular evaluation, monitoring and audit. The Commission did not accept the need to encourage coordinated policy dialogue more than it was already doing. We found evidence of meetings at all levels, as well as joint statements. SR 4/2013 Egypt The Commission said that issues of public finance management and fraud in Egypt would be discussed through informal economic dialogue. We found that a national corruption committee had been set up in line with our recommendation. SR 12/2012 ESTAT The Commission believes that the necessary legal framework and safeguards are in place to ensure that Eurostat’s appointment and dismissal procedures are transparent, ensuring full compliance with the principle of independence. Source: ECA and the Commission’s recommendations, actions and discharge database.

THE COURT’S OBSERVATIONS THE COMMISSION’S REPLIES

CONCLUSIONS AND RECOMMENDATIONS

Conclusions

3.72. Good performance measurement and reporting provides an indication of the success of operations and highlights where improvements should be made. The Commission applies a complex and diverse performance reporting framework consisting of various reporting channels (see paragraphs 3.3-3.12).

3.73. Like other governments and international organisations, the Commission is currently aiming to focus more on performance. We identified six areas of good practice by governments and international organisations around the world in which the Commission could consider improvements (see paragraphs 3.13-3.51): 3.73.

1. Scope for improving the performance framework (paragraphs 3.15-3.23)

— The Commission’s use of two sets of objectives and indicators, for programmes on the one hand and DGs on the other, does not reflect the fact that the Commission is ultimately responsible for implementation of the EU budget with regard to the principles of sound financial management We give examples of governments that have made efforts, by aligning the metrics used, to create a clear framework and increase insight into their performance. The Commission does not report information on expenditure broken down by general and specific objectives. — The ECA observed in its 2015 Annual Report that many of the objectives in the Commission's management plans were not set at the right level of accountability. The Commission has since reviewed its Strategic Planning and Programming cycle. The objective was to provide a clearer framework for the Commission's/Directorates-General's accountability and to make the planning documents more streamlined and centred on the priorities of the Commission and the competencies of the Directorate-General. The objectives and indicators selected for the strategic plans are now tailored to their specific competences and reflect the fact that the responsibilities of Commission departments are broader than budget execution and programme management. For this reason, aligning objectives and indicators in the performance framework of the Commission services with those defined in the legal bases of programmes is no longer required. However, the Commission services are invited to make reference to programme statements when describing their activities in relation to the programmes. This helps to limit duplication of information and ensure consistency of data and will be reinforced in the future. The Commission considers that its responsibility for implementing the budget is clear. Each Director-General signs in his/her Annual Activity Report a declaration of assurance stating that the resources have been used for the intended purpose and in accordance with the principle of sound financial management and that the underlying transactions are legal and regular. At corporate level, the College takes overall political responsibility for the management of the EU budget by adopting each year the Annual Management and Performance Report for the EU budget.

— The Commission uses many different objectives and indicators. While it is undertaking a review under the BFOR initiative, other entities have already gone further towards simplifying their performance framework. — The performance framework of the EU budget has been developed both to fulfil the legal requirements and to respond to the demands from stakeholders and the Commission's own needs. Performance information on the spending programmes is used by the Commission for (a) monitoring programme progress, (b) accountability and transparency, (c) informing decision-making about the management of a programme; (d) justifying or proposing adjustment of the allocation of funds to the budgetary authorities; (e) developing proposals for changes to the legal basis or for future programmes. The Commission notes however that while performance information is taken into account during the annual budgetary procedure, it is not the only factor driving budgetary decisions. More than 700 indicators are included in the legal bases of the 2014-2020 programmes to help monitor their performance during their life-cycle. They are the outcome of the legislative process and therefore reflect the expectations of the co-legislators as regards the type and level of granularity of information required to track performance. Reporting on these indicators in the programme statements enables the budgetary authority to take performance information into account during the budgetary process. Under the EU Budget Focused on Results initiative, the Commission is reviewing the indicators for the spending programmes as established in the legal basis with a view to drawing lessons for the preparation of the next generation of spending programmes. The Commission notes that the number of objectives and indicators used in the strategic plans for the Commission services has been reduced significantly with the recent reform. The number of general objectives has been reduced from 84 to 11, of impact indicators from 187 to 37, of specific objectives from 426 to 386, and result indicators from 969 to 825.

2. Performance reporting is not exhaustive (paragraphs 3.24-3.32)

— Compared with other entities, the Commission reports in a limited way on challenges and failures. Balanced reporting means more clearly analysing past performance in order to make better decisions for the future; it also reinforces the objectivity of the information obtained. — The Commission is committed to producing high-quality performance reports that describe results in a balanced manner. The Annual Management and Performance Report is a high level summary of how the EU budget has supported the European Union's political priorities and the key results that have been achieved with the EU budget. It is not designed to be an exhaustive and detailed report on the performance of the EU budget. The report refers to other performance reports where more detailed performance information can be found (evaluations, programme statements, Annual Activity Reports, etc.). The 2015 Annual Management and Performance Report refers to a number of shortfalls and problems to be addressed. Section 1 of the report refers not only to challenges affecting the timely implementation of programmes but also to limitations in the assessment of programme performance (e.g. references to the evaluations on the 2007-2013 Health programme and programmes in budget heading 4 pointing to constraints in assessing the overall performance of these programmes).

— The limited information on data quality in the Commission’s reports affects transparency. Other entities report more comprehensively on the reliability of performance information. — The Internal Control Framework put in place by each Commission service is designed to provide reasonable assurance with regard to, inter alia, the reliability of financial reporting. Each Directorate-General reports on this aspect under part 2 of its Annual Activity Report, and section 2 of the Annual Management and Performance Report summarises and reports on the assessment by the Directorates-General on the functioning of the internal control systems of the Commission's services. As to evaluations, the Better Regulation Toolbox stipulates that evaluation Staff Working Documents prepared by the Commission services at the end of an evaluation must contain a clear summary of any insufficiencies in the data used to support the conclusions and the robustness of the results. Moreover, the Better Regulation Guidelines stipulate that any limitations to the evidence used and the methodology applied, particularly in terms of their ability to support the conclusions, must be clearly explained in the evaluation reports. This is further strengthened in the Better Regulation Guidelines/Toolbox revision.

3. Performance reports are rather narrative and could make better use of visual and navigation tools (paragraphs 3.33-3.35)

— The Commission is making efforts to create reports that are more attractive to readers and stakeholders. Some other entities focus better on what is essential while making greater use of graphics, tables, colour-coding, infographics, interactive web-sites with navigation tools, etc. — The Commission recognises the importance of using visual aids in its various performance reports and has made improvements in this area. For example, the 2015 version of the AMPR uses textboxes to make key messages and examples more visible and facilitate reading of the report. The 2016 edition of the AMPR has further improved the presentation using graphics, charts and images and including an Executive Summary. This and other reports in the Integrated Financial Reporting Package are accompanied by one-page fact sheets which provide an easily accessible summary of the key information for the reader. Moreover, the instructions for the AARs invite the DGs to use visual elements in the AARs to make those documents more easily readable. The Better Regulation Toolbox includes a dedicated tool on visual aids. This tool aims at presenting some examples of instruments that are considered particularly relevant for policy interventions as a way to raise awareness among Commission services of the added value of such aids. It is by no means meant to bring together all existing visual aids.

4. The Commission does not demonstrate that it systematically uses evaluation results (paragraphs 3.36-3.42)

— Not all evaluations carried out by or for the Commission include recommendations and proposals for implementing actions, as is the case of the best examples from other international organisations and governments. The Commission has not carried out a study, or had one made, on its use of evaluation results since 2005. — The Better Regulation Guidelines foresee that all evaluation Staff Working Documents include findings and conclusions which are the basis for possible follow-up action by the Commission. They also include requirements for the dissemination of evaluation findings and the identification of appropriate follow-up actions. Evaluations by their legal character cannot commit the Commission to any actions directly. A formal report to the Legislator typically sets out follow-up actions under consideration by the Commission. In the 2015 Better Regulation package the Commission committed to assess the functioning of the system before the end of 2019. Preparatory work for this assessment is now starting.

— Unlike some other organisations and governments, the Commission does not have a documented institutional system in place for the regular follow-up of evaluation recommendations, action plans or management responses. — Follow-up to evaluations is an intrinsic part of the Better Regulation system in the Commission whereby evaluation results feed into impact assessments, which are publicly available documents, and the process of annual programming and planning of Commission activities. The Commission follows up on the results of its evaluations in the framework of the Commission Work Programme. In planning the work programme, the Commission systematically reviews evaluation results and determines their follow-up. The follow-up to some evaluation findings are set-out in the REFIT scoreboard which is updated annually and in Commission reports to the legislator accompanying evaluations.

5. Core performance reports do not include a declaration or information on the quality of performance information (paragraphs 3.43-3.45)

— The College of Commissioners and Directors-General do not take responsibility for the performance information they provide. — The overarching principles of the chain of accountability are enshrined in the EU treaties. Within the Commission, following the White Paper on governance, the accountability chain has been defined with a prominent role for the Authorising Officers by Delegation within a decentralised system operating under the political responsibility of the College as Authorising Officer. This is particularly true for the organisation of financial management. At the end of the reporting cycle, the College adopts the Annual Management and Performance Report for the EU Budget and takes overall political responsibility for the management of the EU Budget based on the annual declarations of the Authorising Officers by Delegation. The Commission statement about the responsibility for the management of the EU budget is included to make it clear that the Commission has the ultimate financial responsibility for the management of the EU budget, whereas the responsibility for the results achieved with the EU budget is shared with a wide range of actors at European and national level.

6. Performance information provided by the Commission is not easily accessible (paragraphs 3.46-3.51)

— The Commission produces a vast quantity of performance information. Other governments and organisations group their performance information more rationally, for instance at a dedicated website with a well-developed search engine, user guides and other navigation tools. — Under the EU Budget Focused on Results initiative, a specific EU Results website has been developed. The website is a database which aims to become the single entry point to all EU funded projects — via direct, indirect and shared management. By the end of May 2017, the database has hosted about 1600 EU funded projects in the EU and outside.

3.74. In paragraphs 3.52-3.59, we present common challenges identified in some of our 2016 special reports.

3.75. This year’s follow-up of past recommendations (see paragraphs 3.60-3.71) demonstrated that:

— the Commission accepts and implements a high proportion of recommendations within three years of our audits;

— recommendations that are initially not accepted may later be implemented partially or in full.

Recommendations

3.76. Annex 3.3 shows the result of our review of the Commission’s progress in addressing recommendations on performance issues in our 2013 annual report. That year chapter 10 contained three recommendations. Two were implemented in some respects and one was not implemented.

3.77. Based on our conclusions for 2016, we recommend that the Commission:

— Recommendation 1: streamline performance reporting (58) by

(a) further reducing the number of objectives and indicators it uses for its various performance reports and focusing on those which best measure the performance of the EU budget. In preparing the next multiannual financial framework, the Commission should propose less numerous and more appropriate indicators for the legal framework of the next generation of programmes. In this context, it should also consider the relevance of indicators for which information cannot be obtained until several years have elapsed; (a) The Commission accepts the recommendation. The Commission is currently undertaking a review under the EU Budget Focused on results initiative on the indicators for the spending programmes as established in their legal basis. This will feed into the preparation of the proposals for the next MFF.

(b) presenting financial information in a manner which makes it comparable with performance information so that the link between spending and performance is clear; (b) The Commission accepts the recommendation. It considers that this recommendation is being implemented through the programme statements, which are the instrument through which the Commission justifies the financial resources for spending programmes in the draft budget. For the 2018 draft budget, these include information, for each programme, on financial programming and financial implementation alongside information on performance. It hence considers that the programme statements present information in a way that enable to link spending and performance.

(c) explaining and improving the overall coherence between its two sets of objectives and indicators for programmes on the one hand and DGs on the other. (c) The Commission accepts the recommendation. In the next Annual Activity Report exercises the Commission intends to reinforce this aspect by making greater use of the cross-reference in Annual Activity Reports to programme statements.

— Recommendation 2: better balance performance reporting by clearly presenting information, in its core performance reports, on the main challenges to achieving results. The Commission accepts the recommendation. The Commission is committed to producing high-quality performance reports that describe results in a balanced manner. In the Annual Management and Performance Report the Commission reports on major management challenges in a separate dedicated section (under Section 2). Where problems were encountered in the course of the year the report describes how Commission departments tackled these challenges. In future the Commission will strive to give more information on the main challenges for achieving results in its core performance reports (Annual Activity Reports, Annual Management and Performance Report and programme statements).

— Recommendation 3: further improve the user-friendliness of its performance reporting by making greater use of methods and tools such as graphics, summary tables, colour-coding, infographics and interactive web-sites. The Commission accepts the recommendation. The Commission is committed to continuously improve the user-friendliness of its performance reporting. The 2016 Annual Management and Performance Report has already considerably improved with graphics, charts and images.

— Recommendation 4: better demonstrate that evaluation results are well used by:

(a) requiring evaluations always to include conclusions that can be acted upon, or recommendations, which the Commission should subsequently follow up; (a) The Commission accepts the recommendation. Evaluation conclusions and their follow-up are already an intrinsic part of the better regulation system in the Commission and the process of annual programming and planning of Commission activities. This system is further strengthened with the revision of the Better Regulation Guidelines and Toolbox.

(b) carrying out a new study, or having one made, on the use and impact, including the timeliness, of evaluations at the institution. (b) The Commission accepts the recommendation. In the 2015 Better Regulation package the Commission committed to assess the functioning of the system, including evaluations, before the end of 2019. Preparatory work for this assessment has started.

— Recommendation 5: indicate in core performance reports whether, to the best of their knowledge, the performance information provided is of sufficient quality. The Commission accepts the recommendation. In order to improve transparency, the Commission will provide information on the source and quality of data where available. Given that a significant amount of performance data is provided by Member States, the Commission will analyse to what extent they provide information on the quality of performance data.