THE COURT’S OBSERVATIONS THE COMMISSION’S REPLIES

INTRODUCTION

10.1. This chapter focuses on performance. It is divided into three parts. The first part presents the Court’s observations on the management-plans (MP) and annual activity reports (AAR) of some of the Commission’s directors-general. The second part covers the Commission’s second and third evaluation reports (1). The third part highlights some of the main themes arising and the lessons that can be learned from the Court’s 2012 special reports (2) on performance. 10.1. Measuring and reporting on performance is receiving growing attention within the Commission. Reporting on performance is to be considered as an evolving practise. The Communication ‘Synthesis of the Commission's management achievements in 2012’ (COM(2013) 334 final of 5 June 2013) defines a number of initiatives to further strengthen performance management in the Commission. This progressive development of a performance culture will take several years to come to full effect, partly because new statistical and other tools need to be developed.

10.2. Performance is assessed on the basis of the sound financial management principles (economy, efficiency and effectiveness) (3). Its measurement is key throughout the public intervention process, covering inputs (financial, human, material, organisational or regulatory means needed for the implementation of the programme), outputs (the deliverables of the programme), results (the immediate effects of the programme on direct addressees or recipients) and impacts (long-term changes in society that are attributable to the EU’s action).

10.3. For many areas of the EU budget the legislative framework is complex and there is a lack of focus on performance. Despite the claimed focus on results, the proposals for the new CSF regulations and the common agricultural policy (CAP) regulations for the 2014-2020 programming period remain fundamentally input-based (expenditure oriented) and therefore oriented towards compliance rather than performance (4). For example regarding rural development, a disparate and wide ranging set of objectives is laid down in the regulation which does not include expected results and impacts or relevant indicators (5). 10.3. The new regulations for the CAP do ensure a more performance-oriented approach, also under the second pillar: — the EAFRD draft regulation does not include quantified targets at EU level, but requires that each Rural Development Programme (RDP) should be constructed around the achievement of quantified targets, which are strictly related to the objectives laid down in the regulation. The indicators to be used to define these targets will be laid down in the implementing act; — besides, key planned outputs will be subject to a performance framework as foreseen under the Common Provision Regulation. As regards the proposed Regulations for the European Structural and Investment Funds, resultorientation has been included as essential feature. The demanding and innovative proposals include: — a reinforced intervention logic that links objectives set by the Union with binding and precise objectives set by the Member States; — for the first time intervention-specific ex ante conditionalities to improve the effectiveness of investments; — decisively extended macroeconomic conditionalities, — a renewed approach to evaluation, based on planning, the obligation to evaluate the impact of each part of the programme, and the use of new methods, and — an obligatory performance framework including a performance reserve.

10.4. For large parts of the EU budget, mainly the European Agricultural Fund for Rural Development (EAFRD) and the cohesion funds (CSF), the maximum level of expenditure under the multi-annual financial framework (MFF) (6) headings is broken down into yearly allocations per Member States. As the Court pointed out in the past, it is a challenge to obtain good qualitative results from schemes where funds are pre-allocated among Member States (7) and absorption of these funds is an implicit objective (8). 10.4. The new RDPs will be constructed around the achievement of quantified targets (related to the objectives of the regulation). The budget distribution to the different measures will be in line with these targets, ensuring consistency between the targets to be achieved and the means programmed for them. Besides, a performance reserve will be kept by the Commission, and only released at a later stage in the programming period, when Management Authorities can show that the RDPs have not failed to achieve preset milestones. Thus the prior objective of reaching the targets is strongly promoted.

The budget allocation by Member State is necessary for the purpose of programming, and in particular for the ex-ante quantification of targets which could otherwise not be done. Envelopes should be proportionate to the needs to be addressed in the different Member States.

PART 1 — THE AAR’S REPORTING ON POLICY ACHIEVEMENTS IS EVOLVING BUT IT IS STILL OF LIMITED USE

Introduction

10.5. The various management modes (9) for implementing the EU budget, taken together with the differing roles of DGs, mean that the Commission plays a number of different roles and bears a range of responsibilities. These should be — amongst other things — considered by the Commission when setting performance objectives and defining indicators in their MPs and reporting on them in AARs. 10.5. Commission services have been requested to define in their management plans objectives setting out the expected impacts and results. These objectives usually address societal problems that are being pursued through various means, including financial intervention, regulatory actions or law enforcement. Consideration is being given to ways of improving transparency by better describing in the management plans the responsibilities the Commission fulfils in relation to its operational activities.

10.6. The Court examined the reporting requirements and processes concerning MPs and AARs including the relevant guidance provided and it reviewed the MPs and AARs (part 1) of the Directorate-General for Competition (DG COMP), the Directorate-General for Mobility and Transport (DG MOVE) and the Directorate-General for Maritime Affairs and Fisheries (DG MARE) for the year 2012. In particular, the Court assessed if the AARs provide useful information on the DGs' annual contribution to policy achievements (10). The review was based on requirements established in the Financial Regulation, the Commission's internal control standards and the MP/AAR-instructions issued by the Commission’s Secretariat-General (SG) and the Directorate-General for Budget (DG BUDG).

Reporting on policy achievements needs improvement

10.7. The Financial Regulation requires that the Commission attach activity statements to the draft budget (11) presented to the budgetary authority. These activity statements should include SMART (12) objectives (at least one objective for each activity) for the various activities. These objectives should be measured by indicators. The objectives and indicators are to a large extent directly used by the directors-general as objectives in their MPs.

10.8. The directors-general should then report in part 1 of their AARs on the results achieved and on the extent to which the results have had the impact intended. In the Commission as a whole there are some 1 000 objectives and 3 000 indicators in the MPs taken together (13).

10.9. The Court's review of the MPs and AARs of DG COMP, DG MOVE and DG MARE identified the following main issues affecting the reporting on annual policy achievements: 10.9. The Commission considers that it has made significant progress in recent years as regards reporting on policy achievements and that notably the Annual Activity Reports of its Directorates-General contain a wealth of useful information on the results attained and their impact on society.

(a) objectives taken directly from high level policy or legislative documents are often not sufficiently focused to be useful for MPs and AARs; (a) See Commission reply to paragraph 10.10.

(b) only eight of 52 objectives fulfilled all of the SMART criteria; (b) See Commission reply to paragraph 10.12.

(c) indicators for monitoring performance need improvement;

(d) evaluations have not been a useful source of evidence to substantiate reporting on policy achievements in AARs. (d) Different evaluations have been a useful source of information to support the policy achievements reported upon in Annual Activity Reports. However, not all evaluations directly serve the purpose of the Annual Activity Report as for instance they cover a different time-frame than the year which is reported in the Annual Activity Report.

Objectives directly taken from high level policy or legislative documents are often not sufficiently focused to be useful for AARs and MPs

10.10. According to the 2012 Commission’s standing instructions for management plans, DGs should establish general objectives with impact indicators (long-term) and specific objectives with result indicators (short/medium-term). The objectives used in the MPs and AARs are often taken directly from legal texts, or high level policy documents. The three DGs examined referred to the binding nature of these texts as the underlying reason for this approach. While objectives in such documents should in particular provide the basis when setting general objectives, they are construed in a political context and are often not sufficiently focused to be useful for the purpose at the DG level (see for example box 10.1). In addition, the link between general and specific objectives was not clearly established (DG COMP and DG MARE). A similar conclusion has been made recently by the IAS which reported that for some AARs there is incomplete information on the link between general and specific objectives defined in the management plan. 10.10. The standing instructions for management plans require that the general objectives be aligned with the Commission’s political objectives by further elaborating them in the management plans. The rationale for this approach is given by the need to ensure that the Commission services follow up on the delivery of the political objectives and of the main policy priorities of the Commission. Objectives should focus mostly on impacts and results, even though these are normally only partly attributable to the actions of the services. Being instrumental to ensuring Commission’s public accountability, objectives focus mostly on impacts and results, even in the case these are normally only partly attributable to the actions of the services and are influenced by other external factors. The Commission will consider amendments to the standing instructions for the management plans in order to improve the presentation of the linkages between the general and specific objectives. See also Commission reply to Box 10.1.

10.11. The MPs do not clearly set out how the planned activities at each management level will contribute to the achievement of objectives set, taking into account the allocated resources and the risks identified. This has a negative impact on the usefulness of the reporting on policy achievements in part 1 of the AAR. As such the objectives might be relevant to the policy area within which the DGs operate but are not sufficiently focused to allow for the monitoring of their achievement over time (for example, with milestones). 10.11. The Commission is considering ways to improve the presentation in the management plans of the linkages between the actions the services carry out and the intended results and impacts. The Commission considers that compliance with SMART criteria is sufficient in order to allow for appropriate monitoring of progress towards achieving the objectives. Moreover, impacts and results can often only be measured over a multiannual perspective.

Only eight of 52 objectives fulfilled all the SMART criteria

10.12. DGs' objectives should be clearly defined and updated when necessary. They should be formulated in a way that makes it possible to monitor their achievement. That is to say objectives should be SMART. However, only eight out of the 52 examined objectives fulfilled all the SMART criteria (for one example see box 10.2). While the objectives examined were relevant to the policy area in which the DGs operated they were often not sufficiently specific, measurable or timed to follow-up. 10.12. When assessing the compliance with the SMART criteria, the information in the management plan should be analysed in its entirety. The Commission recognises that performance reporting should focus on the real needs of stakeholders and should be sufficiently flexible to reflect the different nature of activities of the services.

A SMART objective To ‘implement the core transport network by 2030’ is one objective of DG MOVE. This objective is specific, timed and relevant. The measurability of this objective is provided in quantified terms in the MP and AAR (for example as ‘five removed bottlenecks on the core transport routes by 2017’).

Objective not SMART One of the specific objectives presented by DG MOVE in its MP 2012 is: ‘To promote a modern urban mobility’. This objective is not SMART. It is neither specific as it does not provide any information on what needs to be changed nor is it measurable or timed. Objective not SMART For the objective on urban mobility, it will be considered to be attained by the number of cities that are members of the CIVITAS forum reaching 250-270 by 2015. All necessary information is therefore available to the reader and the objective can in practice be considered to be specific, measurable, achievable, relevant and timed, i.e. SMART.

10.13. Milestones make it possible to check whether the achievement of a long-term goal is on track. Long-term targets should be accompanied by medium-term or annual milestones. In its 2012 MP, DG MARE set two milestones which were already achieved in 2011. For other milestones used by DG MARE the relevance to the achievement of the associated objectives was not obvious. DG MOVE used four milestones for objectives with targets in the future (2020 and 2050) in its 2012 MP. However, one of these targets which had a milestone set in 2012 was not reported on in the 2012 AAR without any explanation as to why not. 10.13. Milestones only make it possible to check the extent to which the actions taken are moving towards the achievement of a long-term goal, thereby signalling if progress is advancing as planned. As regards DG MOVE, in the Management Plan 2012 a long-term target (for 2020) and a milestone for 2010 were set for the impact indicator ‘share of renewable energy in transport’. However, the legislation on which the milestone was based was repealed with effect from 1 January 2012 and thus the corresponding indicator was no longer reported in the AAR 2012. The Commission agrees that an explanation in the AAR 2012 would have been useful in clarifying why the level of the milestone for 2010 is included in the Management Plan 2012 but not in the AAR 2012.

Indicators for monitoring performance need improvement

10.14. The choice of indicators should be based on considerations such as the relevance, measurability, and the availability of timely and reliable performance data. To the extent possible, such indicators should be established according to the ‘RACER’ criteria (14). Indicators, which are measuring outcomes over which DGs have limited influence, should be accompanied by other indicators measuring directly the DGs' activities. 10.14. The indicators should be relevant for the objectives they relate to. Where objectives are impact — or result-oriented, indicators measuring only the direct outcome of the DG are too restrictive to adequately monitor progress towards achievement of those objectives. The Commission considers that using only indicators measuring a direct outcome of a DG would be overly restrictive and not adequate to cover the wealth of its activities.

10.15. All three DGs have at least one performance indicator per objective to help management monitor, evaluate and report on achievements. An examination of 15 indicators (five for each DG) showed that eight of the indicators adequately fulfilled all the RACER criteria. In addition, the Court identified the following shortcomings (see box 10.3): 10.15. The Commission considers that the indicators presented generally provide useful information on progress towards attaining the Commission’s overall policy objectives as well as for monitoring performance.

(a) indicators which were outside the DGs' ability to influence (DG MARE, DG MOVE); (a) The Commission considers that the indicators present important information for communicating with external stakeholders and for providing an overall steer to the political initiatives and decisions of the Commission. The direct impact of a policy initiative on the attainment of an objective can often only be assessed in a medium or longer term perspective.

(b) indicators which lacked relevant information (DG COMP) or a quantified target (DG MARE and DG COMP) or where a quantified target was available it could not be measured (DG MARE); (b) Please see reply to Box 10.3.

(c) indicators with targets where the link between the indicator and the target was unclear or the target was not meaningful (DG MARE and DG MOVE). (c) The Commission considers that the indicators presented are useful. It nevertheless agrees that information on the attainment of the targets for these particular indicators should be more clearly explained and will strive to do so in the future (see also reply to Box 10.3).

DG MARE — Indicators outside the control of the DG There are indicators which fall outside of DG MARE’s scope of influence as they rely on results of international trade negotiations. DG MARE mentions as a disclaimer that completion of these negotiations is not under its control. DG MARE — Indicators outside the control of the DG The indicator on international trade negotiations under ABB 1102 is relevant for measuring progress towards the specific objective concerned, as it indicates the result of the work of the DG. A disclaimer has been added in accordance with Part 4 of the Standing instructions from the central services on the Management Plan 2012.

DG COMP — No relevant information provided The specific objective ‘Effective detection, sanctioning, deterrence and remedying of the most harmful anti-competitive practices by undertakings other than cartels’ is measured by one result indicator ‘Benchmark for the (observable) customer benefits from Commission decisions prohibiting anti-competitive practices other than cartels and from Commission decisions making binding the commitments put forward by undertakings’. This indicator is defined according to a calculation methodology used by DG COMP which aims at quantifying estimated consumer benefits. However, DG COMP itself states that this indicator does not measure progress towards the achievement of the objective. Hence, the objective is without an indicator. DG COMP — No relevant information provided The Commission acknowledges that DG COMP does currently not provide for an indicator for its activity relating to anti-competitive practices by undertakings other than cartels. This is due to the complexity of the calculation due to divergence of such cases compared to the more homogenous nature of the cases for instance for horizontal mergers or cartels implying a less complex calculation, and the incurring resource implication for such more complex calculation (cost-benefit).

DG MOVE — Difficulties with presenting and interpreting its results For the general objective ‘modernise transport infrastructure and its smart funding’ one of the impact indicators established by DG MOVE is ‘reduction in (average) travelling time on 30 TEN-T priority projects due to improved infrastructure financed through EU funds’ with the target of minus 20 % by 2020 (in comparison to 2011). The impact on the population is stated in quantitative terms as reduced travelling time. However, the way the latest known information is presented in the 2012 AAR is confusing. The current situation for this indicator is presented as follows: ‘St-Pölten-Vienna: 40 min; Unterinntal: 15 min; Barcelona-France: 1 h 40 min’. It is unclear if this is the reduced average traveling time already achieved in 2012 or indeed the reduced average traveling time target for 2020. There is no information to what extent the target of the minus 20 % was achieved. In addition only three values out of 30 are presented. Without additional explanation it is not possible to determine whether the realisation of the objective as monitored by the indicator is on track. DG MOVE — Indicator for reduction of travelling time The Commission considers the indicator for the reduction of travelling time to be very relevant and particularly useful for the assessment of the achievement of the objective to modernise transport infrastructure. It however acknowledges that the progress on achieving the reduction of travelling time on 30 TEN-T priority projects (PPs) could have been more clearly explained and will strive to do so in the future. On the stretches of the three priority projects mentioned, travelling times were shortened by 20-60 % in 2012 due to measures co-financed by the EU. As interventions on further sections will follow until 2020, the Commission expects that individual objectives will be met also for the entire stretches of these priority projects, as well as for other priority projects.

Evaluations have not been a useful source of evidence to substantiate reporting on policy achievements in AARs

10.16. According to the Commission’s standing instructions, evaluations and evaluation related studies should be a source of information on results and impact. All three DGs include some information on evaluations in part 1 of their 2012 AARs. However this information was brief or in footnotes. An examination of five evaluations completed in 2012 or earlier for each of the three examined DGs revealed that these were of limited value as sources of information on results and impact. This was largely due to the fact that evaluations were oriented towards operational questions rather than performance or because the Commission had doubts as to the quality of information obtained from Member State authorities. 10.16. The Commission takes note of the Court’s finding that the sample of evaluations, looked into by the Court, were oriented towards operational questions rather than performance. The Commission shares the view that evaluations should produce information on results and impacts, recognising that the focus of evaluations may differ depending the stage of implementation of the evaluated programme. The Commission refers to the Article 318 TFEU evaluation report in which results, covering all of the MFF Budget headings, are presented in relation to both operational issues during the early implementation of programmes as well as in relation to the impact, effectiveness, and EU value added of programmes at a later stage of implementation. The Commission will pay increased attention to performance-related aspects in its evaluations and to the necessary reporting on this in the AARs. It points to examples of evaluations already providing useful information on policy achievements. The Commission considers that it has made significant progress in recent years in reporting on policy achievements and that the Annual Activity Reports of its directorates-general contain a wealth of useful information on policy objectives, the results attained and their impact on society. The Commission is working on a stronger and more coherent framework for monitoring, evaluation and reporting on the performance of EU financial programmes for the next MFF. The Article 318 Report on the year 2012 contains an Action Plan. The Commission recalls that the information and assessments that the Commission will be able to produce year-on-year in its Annual Activity Reports and in its overall reporting for the next MFF will largely depend on the provisions regarding performance in the legal instruments to be adopted by Parliament and Council to support the new set of financial programmes.

Performance management and reporting is evolving and some corrective actions have been taken but it is too early to assess their results

10.17. SG and DG BUDG play a central role in providing annual instructions concerning the structure and content of AARs. In addition they exercise a central quality management (support and monitoring) over AARs prepared by the individual DGs. The quality control process included, for the first time in 2012, discussions of the draft AARs part 1 of all DGs with SG and DG BUDG.

10.18. In its audit of the AAR process, the IAS concluded that the process focuses on legality and regularity while economy, efficiency and effectiveness of policy achievements are not sufficiently addressed. In response to this audit the Commission established an action plan in March 2013 to address the main findings including a revision of the standing instructions. 10.18. The Commission will revise the standing instructions on the preparation of the Annual Activity Reports. This revision will also ensure that economy, efficiency and effectiveness of policy achievements are addressed.

10.19. In the context of the MFF negotiations the Commission has proposed a common performance framework for both the Commission and the national authorities which bases itself on agreed output and result indicators.

PART 2 — EVALUATION REPORT ON THE UNION’S FINANCES BASED ON RESULTS ACHIEVED

10.20. The Commission published its second report in November 2012 and in June 2013 its third report on the evaluation of the Union’s finances based on the results achieved (the evaluation report) (15). The Treaty requires that the Commission produce such a report, and that the report is part of the evidence on which the Parliament gives a discharge each year to the Commission in respect of the budget (16).

10.21. In its observations (17) on the second evaluation report the Court found the broader scope, compared to the first evaluation report, an improvement and noted that the report gives some indication as to the effectiveness and efficiency of the programmes. However, the evaluation report did not provide sufficient, relevant and reliable evidence on what the EU’s policies have achieved and was therefore not suitable for its intended use in the discharge procedure. 10.21. The Commission reiterates its intention to further improve the report in the future in discussion with the Discharge Authority. The Commission also confirms that this can only be achieved under the new performance framework which will be based on the legal bases for the next MFF. Account has been taken of the indications provided so far by the Court in the preparation of this year’s evaluation report to the extent possible within the existing framework of the MFF 2007-13. This report was adopted on 26 June 2013.

10.22. Compared to the first and the second evaluation reports, the third report represents some improvements as it draws on more sources of information than just evaluation reports, notably the Court's special reports and recommendations of the Commission's Internal Audit Service, and also identifies some areas in need of improvement. The Court welcomes these developments. However, the evaluation report is not yet suitable for its intended use. 10.22. The Commission welcomes the positive reception by the Court of the developments made in the evaluation report. At the same time, the Commission considers that it may only be possible for the evaluation report start to provide sufficient, relevant and reliable evidence on what EU policies have achieved, and therefore to be considered by the Court as suitable for its intended use in the discharge procedure, late in the MFF when the first indications on the impacts of the programmes start to appear.

10.23. The Court notes that the Commission has undertaken to present, in 2014, a new framework for performance reporting and notes the initiatives already taken in order to strengthen performance reporting in the Commission services. In addition, the Commission plans to: (a) improve performance reporting in the annual activity reports of its directorates-general; (b) make a closer link between the annual management plans and annual activity reports on the one hand and the Article 318 evaluation report on the other hand; (c) adopt the evaluation report in parallel with the synthesis report. 10.23. On the basis of the new generation of spending programmes under the multiannual financial framework for the period 2014-2020, the Commission will instruct its services to reflect the performance framework of these programmes into their management plans for 2014. This will allow reporting on performance in the Annual Activity Reports to be issued in March 2015. In addition, a closer link will be created between the Annual Activity Reports of the directorates-general and the evaluation report based on Article 318 TFEU.

10.24. Finally, the third evaluation report is accompanied by an action plan for developing the Article 318 evaluation report. The Court recognises this as a positive step. However, it is not clear what the actions will mean in practice. In particular, the plan does not specify individual responsibilities or completion dates for the actions mentioned. The Court will follow-up and report on the extent to which the Commission succeeds in implementing the action plan and in delivering the improvements promised concerning future evaluation reports. 10.24. The Commission welcomes the Court’s recognition that the Action plan is a positive step. It recalls the indication in this year’s report that the Action plan remains provisional pending the adoption of the legal instruments that will support the financial programmes in the next MFF. This explains the absence of specific individual responsibilities and completion dates.

PART 3 — RESULTS OF THE COURT’S AUDIT ON PERFORMANCE

Introduction

10.25. The Court’s special reports examine whether the principles of sound financial management (paragraph 10.2) are applied to European Union spending. The Court chooses its topics for special reports — specific budgetary areas or management themes — to have maximum impact, based on a range of criteria, such as the level of income or spending involved (materiality), the risks to sound financial management and the degree of stakeholder interest. In 2012, the Court adopted (18) 25 special reports, as listed in Annex 10.1 .

10.26. The Court examines in this part the lessons that can be learned from its 2012 special reports. The Court focuses on three themes which are important for obtaining the desired results and impacts of the next generation of spending programmes: SMART objectives and suitable indicators for programmes, reliable and timely data on the performance of programmes and sustainability of EU funded projects.

10.27. These themes are also reflected, to a varying degree, in other recent publications from the Court (19). As previously pointed out by the Court, and in order to establish whether European Union spending is having the desired results and impact, there is a need for the Commission to re-think its reporting and accountability systems for the forthcoming programming period and focus not only on compliance with the rules but also on the achievements of outcomes (results and impact). SMART objectives and suitable indicators along with reliable, timely information, are essential elements of such a reporting and accountability system. 10.27. The Commission agrees that its management systems should be adequately set in order to measure the achievements and outcomes of the EU spending and that appropriate follow-up and control systems should be in place to obtain reliable information for reporting on results. The Commission is working on developing its internal management system in the context of the upcoming 2014-2020 multiannual financial framework (MFF) with a particular focus on EU added-value, efficiency and effectiveness and overall impacts on the economy and society of its spending programmes. The Commission will design a new culture of performance using the objectives, indicators, monitoring and evaluation frameworks set by the Legislative Authority for the coming programming period; this will also provide a clear basis for the annual reporting on performance by the Commission. The Commission has already included in its proposals for the new programmes under the MFF several elements aimed at delivering a stronger performance framework, which would be more focused on efficiency and effectiveness in the attainment of the overall objectives of the different financial programmes. This performance framework will impact on the existing internal management instruments.

SMART objectives together with suitable performance indicators not consistently applied in spending programmes

10.28. The Financial Regulation requires that the achievement of SMART objectives set for all sectors of activity covered by the budget be monitored by performance indicators (20). This would allow the Commission and Member States to evaluate not only at the end but also at key intermediate points whether the defined objectives have been met and whether intended impacts are likely to be achieved.

10.29. In its special report on integration of third-country nationals, the Court found that the Commission has demonstrated that it is possible to define indicators relating to the output and outcomes of funded actions (21). The examined authorities in Member States were required to include quantifiable outputs. However, measurable targets were only specified for around half of the actions examined and most of these cases referred simply to the number of projects or participants instead of specific aspects such as the success rate in obtaining qualifications. The Court concluded that although the Commission provided extensive guidance on indicators, four out of the five Member States showed significant weaknesses in the inclusion of SMART objectives in programmes, setting up appropriate IT collection systems and/or reporting on the achievement of targets. Consequently, the ability of Member States and the Commission to monitor and steer the programmes has been hampered. 10.29. Under the European Fund for the Integration of third country nationals, the Commission is using several instruments to monitor national programmes, such as regular contacts with Member States and intermediate and closure implementation reports. In addition, Member States have submitted national evaluation reports covering the period 2007-2010 on the basis of common indicators set by the Commission and compatible with the Member States’ own indicators at project level. These common indicators should produce meaningful results at national and European level. In its proposal for a Regulation for the Asylum and Migration Fund of 15 November 2011 (COM(2011) 751 final), the Commission has proposed to have common compulsory indicators to assess and to manage efficiently the national programmes. They are already being discussed with participating Member States, in particular as regards the fact that they should be limited in number and ‘SMART’, linked to reality and providing reliable data.

10.30. The special report on the reform of the wine market showed that despite thorough information requirements and the Commission receiving a large amount of information from Member States on the implementation of the various measures, the Commission’s monitoring role, which should include an assessment of whether the reform is meeting its objectives, was limited by the absence of key performance indicators (22). 10.30. The assessment of whether the wine Reform of 2008 meets its objective is carried out by the Commission on the basis of the following elements: (1) Periodical assessment of the wine market situation: (a) based on key indicators provided for in Regulations (EC) No 555/2008 and (EC) No 436/2009. (2) Periodical assessment of the implementation by Member States of wine policies for example: (a) a study on the implementation of the reform carried out in 2012, which provided a substantial and detailed analysis on different aspects of the wine common market organisation; (b) as regards national programmes, Regulation (EC) No 555/2008 was modified and two guidelines were respectively approved in 2013 by DG AGRI with the view to clarifying and detailing how these measures have to be implemented at national level. These guidelines recommend to Member States key indicators to be used (e.g. selection criteria, evaluation criteria, impact indicators, etc.). These indicators are also used by DG AGRI when evaluating its policies. (3) Broader evaluations of wine issues: (a) the data collected throughout 2012 was also used by DG AGRI for the drafting of the 2012 report to Council and Parliament, in combination with other indicators provided for by Member States relating to national support programmes and to the grubbing-up scheme; (b) DG AGRI is currently launching a new evaluation on the competitiveness of EU wines, which should be completed by the end of 2014. The proposed tender establishes clear indicators for this up-coming evaluation; (c) DG AGRI is carrying out another evaluation on the added-value of protected designations of origin and protected geographical indications. Again, this later establishes clear indicators.

10.31. The Court’s special report on financial instruments co-financed by the ERDF points out that the standard cohesion policy monitoring instruments put in place for the ERDF are inadequate or not adapted for the purpose of financial instruments. It concluded that where they existed, monitoring and information systems were ill-equipped to provide information on and monitor the sound financial management of the funds (23). 10.31. The amendment of Council Regulation (EC) No 1083/2006 in December 2011 introduced an obligation for Member States to formally report on financial instruments. In 2012, the Commission produced on this basis a summary report and the Commission services have also developed a detailed guidance on reporting for managing authorities. In addition, in February 2012 the Commission published a Staff Working Document on financial instruments in cohesion policy and carried out an evaluation through the expert evaluation network on the use of the ERDF to support financial instruments. The reporting requirements will be further consolidated in the next programing period.

Deficiencies in the relevance, reliability and timeliness of performance data

10.32. Measuring and assessing achievement of objectives is however only meaningful if based on information which is relevant, reliable and timely. In building a robust performance management and reporting system, the Commission needs to give particular attention to the reliability and timeliness of the performance data that it generates or obtains from Member States and third parties. A number of the Court’s 2012 special reports provided examples of deficiencies in the quality of data and some examples are given in the following paragraphs. 10.32. For the programmes in the area of direct management, the Commission should be able, according to its legislative proposals for the next funding programmes to ensure timely and reliable information for reporting on their performance. For the programmes under indirect management the Commission has to rely on the willingness and capacity of Member States and third parties for providing adequate, timely, reliable and comparable data. This is not possible across all the programmes in a synthetic way under the current legal framework. For the programmes under shared management, the Commission proposed to the Legislative Authority some key requirements for the design of the systems to produce information on results in the frame of the future regulations post 2013. The responsibility for operating those systems in order to provide complete and accurate information on results lies with the Member States in line with the division of competences under shared management.

10.33. In its audit on targeting the aid for modernisation of agricultural holdings, the Court found that the Common Monitoring and Evaluation Framework — set up as a tool for the Member States and the Commission — did not generate relevant data which can be used for monitoring the results obtained with the funds spent. The indicators defined did not allow progress in the achievement of the EU’s priorities to be monitored, and the data were found to be of low reliability and did not allow comparisons between Member States (and/or regions) to be made (24). In another report, the Court noted significant weaknesses regarding the reporting by Member States, resulting in the inability of the Commission to evaluate in all cases whether the performance being achieved by Member States was in line with EU waste targets. While the Commission checked the consistency of the figures received, it did not assess their reliability (25). 10.33. The Commission considers that the Common Monitoring and Evaluation Framework constitutes a relevant and integrated set of indicators that was jointly prepared by the Commission and Member States to capture the policy’s objectives. It marks the first time that a comprehensive, yet very demanding, monitoring and evaluation system for rural development was implemented at EU-level. It is the combination of all the measures which should be considered when assessing whether EU’s priorities are well addressed. Measure 121 should be assessed against the objective of improving competitiveness. The current ‘learning by doing’ experience shows that the definition of result indicators in this field is challenging and the Commission, Member States and various stakeholders have devoted much effort to continuously improve methodology, reliability and comparability. This experience will also be used for the future monitoring and evaluation system.

10.34. Shortcomings in reporting systems can also have a negative impact upon the reliability of information. The audit of the Common External Relations Information System (CRIS) — the information system put in place by the Commission to support the financial and operational aspects of the management of external actions — identified problems with regard to the integrity of the data in this system. The Court found records where information was missing, not valid or not up-to-date. These undermined the system’s reliability and its efficiency and effectiveness as a management tool (26). 10.34. Efforts are under way to address those shortcomings, but no financial and/or other critical information is missing in CRIS. In addition to the progress that was mentioned in the CRIS audit as regards data quality issues (nomination of data owners, creation of a data quality team and implementation of a data quality governance), the following projects have recently been launched: — rationalisation of geographical codes, — improvement of CRIS user interface on the basis of the analysis of encoding error cases, — a number of cleansing actions, resulting from audit findings, monitoring procedures and reporting exercises. The above projects are part of a newly defined data quality master plan including actions covering data cleansing, information system improvement, monitoring and control, user documentation and training.

10.35. The audit of the suckler cow and ewe and goat direct aid also showed that monitoring of performance indicators was weak in most Member States due to the lack of an ad hoc performance monitoring system for measuring the results and outcomes of the audited schemes, and due to the different interpretations of the terms ‘production’ and ‘supply to the processing industry’ by each Member State. An additional constraint in the monitoring by the Commission was the lack of an appropriate tool providing a global view of the aids that are available from different sources. In addition, the rapid pace of recent CAP reforms did not always enable precise quantitative assessments to be made in a timely manner (27). 10.35. The legislator has considered that these concepts could be better defined at national level. The Commission has an overall view of the situation across the EU regarding the various coupled support measures applied in the animal sectors. Member States are subject to the obligation to periodically notify the Commission on the number of animals and the amounts paid under both coupled suckler cow and sheep and goat payments and coupled Article 68 payments, as well as under complementary national direct payments (CNDP) implemented in the Member States who joined in 2004 and 2007. As far as social, economic and environmental impacts of the schemes, the Commission considers that it is more a matter for evaluation than monitoring. When it comes to the decision making process for the upcoming reform, the Commission bases its proposals on an impact assessment which draws on a whole range of information sources, quantitative analyses, qualitative and quantitative information from the literature and public consultations’ evaluation reports are one of these information sources. As for the future, the Commission proposal for the CAP towards 2020 foresees the establishment of a common monitoring and evaluation framework with a view to measuring the performance of the common agricultural policy, including first and second pillar measures.

10.36. Timeliness when reporting performance data has proven to be an issue as the shortcomings in the collection and registration of the monitoring data for the European Social Fund spending on older workers demonstrates. Not having access to reliable, verifiable and timely performance data and assessments of the different actions impeded the drawing of conclusions for current and future decision making (28). 10.36. The current regulatory framework for ESF imposes the collection of data at priority axis level, not at the far more detailed target group level. The suggested line would be very complex to implement (for example most projects include de facto many different target groups), place an important burden on Member States and beneficiaries, and entail a considerable cost. Information on results and in particular impact can be obtained through evaluations. For example, the Evaluation of the ESF support to Lifelong Learning, finalised in July 2012, focused inter alia on older workers. The Commission estimates that data available through such evaluations are vital and allow drawing conclusions on the effectiveness of ESF interventions. Nevertheless, the Commission acknowledges that improvements can be made. Therefore, for the next programming period 2014-2020, the Commission aims to address, through the new legislative package, a number of weaknesses identified in the domain of monitoring and evaluation arrangements.

In several cases sustainability of EU projects was at risk

10.37. A public intervention should ideally be sustainable i.e. it should not only focus on short-term outputs but also continue to deliver results and achieve impacts after the external funding has stopped. Sustainability should therefore be an important factor to consider when designing a programme and/or project. Considerations in this regard should include issues such as financial sustainability, technical sustainability and institutional capacity, as well as the interdependence of these elements.

10.38. Eight reports identified sustainability as an issue in 2012. Five of these eight reports related to the assistance provided to non EU countries and territories. The remaining three reports touch upon the sustainability of projects financed under shared management thus providing an indication that this subject is relevant beyond the development aid area. Meanwhile, in six of these eight reports a risk was identified that projects might cease and benefits might not continue to flow in the absence of continuous funding (29).

10.39. Under shared management, the Member State authorities are primarily responsible for selecting projects which are financially sustainable. The Court’s audit of EU support to the implementation of hygiene legislation in slaughterhouses showed that project sustainability was an objective as funded projects should be maintained at target levels of activity for at least five years after the end of the investment. However, the audit also showed cases in each of the five Member States covered by the audit where either the five year minimum period of activity or where the targeted levels of activity were not achieved. This diminished the impact of the funds used to facilitate compliance with hygiene requirements in the slaughterhouses (30). 10.39. The impact of the funded investments on meeting the new hygiene standards was subject of a review by the ex-post evaluation reports of the Sapard programmes of eight beneficiary countries which joined the EU in 2004. The synthesis report of these evaluations by independent evaluators concluded under Chapter 4.2.1 ‘Meeting EU standards and acquis communautaire: The most important impacts were reached in the field of hygiene, sanitary and veterinary conditions and environment’. The report was published and can be found under: http://ec.europa.eu/agriculture/evaluation/rural-development-reports/index_en.htm

10.40. In its audit of development aid for drinking water supply and basic sanitation in sub-Saharan countries the Court found that projects examined were technically sustainable: they promoted the use of standard technology and locally available materials. However, for a majority of projects results and benefits will not continue to flow in the medium and long-term unless non-tariff revenue can be ensured. Despite comprehensive management procedures, the Commission did not tackle important matters regarding sustainability (31). 10.40. To address institutional sustainability issues, the Commission supports extensive capacity-building and training activities to empower water user associations and local authorities to own, manage and operate their installations, sometimes with the support of the private sector. The Commission also works at country level in the framework of sector reform programmes to support the development of appropriate sector policies and institutional framework whereby decentralised structures are set up to help local stakeholders in the long-run. Establishing the water sector on a financially sustainable basis requires establishing the appropriate balance between tariffs, transfers and taxes. For most countries, operating costs cannot be covered by tariffs only. This is true in Europe and even more in sub-Saharan countries, where projects mostly target the most vulnerable groups with very limited capacity to pay. For this reason, the Commission supports the process of developing social tariffs while ensuring the financial sustainability of the sector as a whole through sector dialogue with beneficiary countries.

10.41. In its audit of brownfield site regeneration, the Court observed that all the brownfield site regeneration projects examined had some of the key characteristics which are considered to be best practices for assuring the sustainability of sites and outcomes in the long-term (32). For example, some sites were strategically located and were served by tri-modal transport networks (road, rail and waterways) and all sites were adequately served by transport, sanitation and information and communication technology infrastructure. 10.41. The Commission welcomes the Court’s analysis.

10.42. Even when interventions are financially and technically sustainable, results might not be long-lasting if beneficiaries do not get involved and develop a sense of ownership or where the institutional capacity is insufficient.

10.43. Through both the Instrument for Pre-Accession Assistance and its Common Security and Defence Policy, the EU provides support to Kosovo to strengthen the rule of law. Political support from the Kosovo authorities is a key condition for the sustainability of the results. However, the national authorities' commitment to the rule of law is open to question. Overall, the durability of results which have been achieved by the assistance is threatened by a lack of political will, weak financial capacity and the limited influence of civil society (33). 10.43. The Commission and the EEAS share the Court’s assessment. They aim to remedy the weaknesses identified through the increased use of policy dialogues as recommended by the Court, including with civil society, as well as by better coordination and synchronisation of EU policy objectives, instruments and assistance. This should help make EU action more efficient, effective and sustainable.

10.44. Similarly, the audit of the European Union assistance to the Turkish Cypriot community also showed that sustainability was at risk. This was due to limited administrative capacity, the delayed adoption of relevant measures and uncertainties over future funding (34). 10.44. The Commission has integrated measures into the 2012 and draft 2013 programmes to ensure completion and sustainability of existing projects. This includes continued training and assistance on setting up operation and management systems and also complementary investments to insure the Turkish Cypriot community (TCc) is properly equipped to deal with take-over and running of new plant. A very cautious approach is taken to new investments and is suspended in some areas pending efficiency actions by the TCc. The delayed adoption of relevant procedures and legal texts by TCc has been identified by the Court of Auditors as a risk to sustainability. In these cases, the Commission continues to encourage the TCc to continue their work and delivers support in relevant areas. To address the uncertainties over future funding, the Commission has prepared a proposal for a multiannual perspective under the 2014-20 MFF. This should help to provide a more stable level of funding for operations under the Aid Regulation.

CONCLUSIONS AND RECOMMENDATIONS

Conclusions

10.45. For many areas of the EU budget the legislative framework is complex and there is lack of focus on performance. Proposals for the new CSF regulations and the CAP regulations for the 2014-2020 programming period risk remaining fundamentally input-based (expenditure oriented) and therefore oriented towards compliance rather than performance (see paragraphs 10.3 to 10.4). 10.45. The Commission shares the Court’s views that the current legislative framework is complex and lacks focus on performance. This is why it has made all efforts that its proposals for the new generation of programmes under the multiannual financial framework 2014-2020 contain a strong performance framework, based on a clear definition of SMART objectives, using appropriate indicators allowing to report not only on the outputs but also on the results and impacts of spending programmes. It very much hopes that this framework will be maintained in the legislative process currently on-going between the European Parliament and the Council in view of the final adoption of the legal bases. The new regulations for the CAP include important new elements to ensure a more result-oriented approach (see reply to point 10.3): — the new direct payments have been designed to address specific needs and objectives (environmental objectives, young farmers, areas with natural constrains, etc.), — the EAFRD draft regulation requires that each rural development programme (RDP) should be constructed around the achievement of quantified targets, which are strictly related to the objectives laid down in the regulation. If a RDP manifestly fails to achieve preset milestones, a performance reserve will not be released (see reply to point 10.4). As regards the proposed Regulations for the European Structural and Investment Funds, result-orientation has been included as an essential feature. The demanding and innovative proposals include: — a reinforced intervention logic that links objectives set by the Union with binding and precise objectives by the Member States, — for the first time intervention-specific ex ante conditionalities to improve the effectiveness of investments, — decisively extended macroeconomic conditionalities, — a renewed approach to evaluation, based on planning, the obligation to evaluate the impact of each part of the programme, and the use of new methods, and — an obligatory performance framework including a performance reserve.

10.46. In the context of the MFF negotiations the Commission has proposed a common performance framework for both the Commission and the national authorities which bases itself on agreed output and result indicators. However, an important factor for the future success in this area is its successful implementation by the Commission and the Member States (see paragraph 10.19). 10.46. The Commission notes that for all programmes under shared management, the responsibility for implementing the common performance framework lies primarily with the Member States which are responsible for the daily management of the programmes, their monitoring and the reporting on results.

10.47. The examination of the three DGs' AARs leads to the conclusion that AARs' reporting on policy achievements is evolving but it is still of limited use. The Court identified shortcomings in objectives and indicators as main issues (see paragraphs 10.9 to 10.16). The observations in the corresponding chapters in the Court’s two previous annual reports are similar to this year. 10.47. The Commission considers that it has made significant progress in recent years as regards reporting on policy achievements and that notably the Annual Activity Reports of its Directorates-General contain a wealth of useful information on policy objectives, the results attained and their impact on society. The indicators presented generally provide useful information on progress towards attaining the Commission’s overall policy objectives as well as for monitoring performance.

10.48. The Article 318 evaluation report continues to be a work in progress and is not yet suitable for its intended use in the discharge procedure. The Court recognises the Commission’s commitment to strengthen the performance culture within the Commission and to develop a reliable and robust performance management and reporting system. Towards that end, the Commission will present, in 2014, a new framework for performance reporting for the next MFF. Furthermore, the Commission has taken initiatives to strengthen and streamline its services reporting on performance in the remainder of the current programming period (see paragraphs 10.20 to 10.24). 10.48. The Commission recognises that the report continues to be a work in progress, with a strong focus each year on possible improvements for each following year. However, it doubts that the report can fulfil any intended use in the discharge procedure to start to provide reliable evidence on what EU policies have achieved before late in the MFF.

10.49. As in previous years, in its performance audit work published in its 2012 special reports, the Court has identified weaknesses affecting the entirety of a typical management lifecycle. In the three specific areas analysed this year, the Court found that there are weaknesses in essential elements of the performance management and reporting system: SMART objectives and suitable indicators are not consistently employed in spending programmes, there are deficiencies in the quality and timeliness of performance data and the sustainability of projects financed by EU spending is often not ensured. It is important to address these issues if the desired results and impacts are to be obtained from the next generation of spending programmes (see paragraphs 10.25 to 10.44).

Recommendations

10.50. Annex 10.2 shows the result of the Court’s review of progress in addressing recommendations made in the 2010 annual report. In the 2010 annual report (chapter 8), the Court presented six recommendations. All six recommendations were implemented in some respects. This assessment of the progress is based on: 10.50.

(a) some positive developments in central guidance and quality management (for example improved instructions for the 2013 MPs and improved quality control of part 1 of the 2012 AARs);

(b) the remaining substantial shortcomings in the AARs examined; and (b) The Commission is pursuing its efforts to improve the Annual Activity Reports. These will aim to: — include more elements of performance management: setting objectives, performance indicators and associated targets at the level of each programme, — deepen the reporting on how financial and human resources have been used to achieve the policy objectives set by the College, and on how these policies have generated EU added value, — strengthen the internal peer-review process on the Annual Activity Reports, including increased focus on the performance information.

(c) the weaknesses identified in the Court’s special reports.

10.51. Following this review and the findings and conclusions for 2012, the Court recommends that: 10.51.

— Recommendation 1: the Commission and the legislator should ensure that there is a focus on performance in the forthcoming programming period (2014-2020). This requires that a limited number of sufficiently specific objectives with relevant indicators, expected results and impacts are laid down in the sector-specific regulations or in some other binding manner; The Commission proposals for the programmes under the next multiannual financial framework focus on delivering on key policy priorities and on performance. They included elements of an enhanced performance framework in the form of provisions on objectives and indicators, monitoring arrangements, evaluation and reporting requirements. However, under the ordinary legislative procedure, both the Council and the European Parliament have the final responsibility for the adoption of the legal basic acts.