THE COURT’S OBSERVATIONS THE COMMISSION’S REPLIES

INTRODUCTION

8.1. This Chapter presents the Court’s observations on the Commission’s self-assessment on performance as stated in the Annual Activity Reports of the Commission’s Directors-General (1) (2) and the main performance audit results for the last financial year as presented in the Court’s Special Reports (3). 8.1. The Commission welcomes the audit undertaken by the Court on performance information, at a time when increased attention is put on the need to demonstrate the added value and the impact of EU spending.

8.2. Performance is assessed on the basis of the sound financial management principles (economy, efficiency and effectiveness) (4). Its measurement is key throughout the public intervention process, covering inputs (financial, human, material, organisational or regulatory means needed for the implementation of the programme), outputs (the deliverables of the programme), results (the immediate effects of the programme on direct addressees or recipients) and impacts (long-term changes in society that are, at least partly, attributable to the EU’s action). 8.2. Measuring the effects of complex policies is not always easy, given the range of contributing factors, as well as the combined effects of integrated programmes. Monitoring captures what can be observed (and reported, among others, through the Annual Activity Reports), and evaluation is needed to determine effects due to the policy.

THE COMMISSION’S SELF-ASSESSMENT ON PERFORMANCE

Introduction

8.3. The Directors-General of the Commission set performance objectives in the yearly Management Plans of the Directorate-General and then report on the achievements in the Annual Activity Reports.

8.4. The Court assessed the performance information provided in the Annual Activity Reports of the Directorates-General for Agriculture and Rural Development (DG AGRI), for Regional Policy (DG REGIO) and for Research and Innovation (DG RTD) (5). These are respectively the two largest areas of shared management expenditure and the largest area of expenditure directly managed by the Commission. The Court focussed particularly on objectives and related performance indicators, assessing their relevance (6), comparability (7) and reliability (8). 8.4. The Annual Activity Report (AAR) is only one of the tools to report on performance. The Commission has reinforced its evaluation capacity and useful conclusions are presented and explained in evaluation reports (see reply to paragraph 8.23) carried out by all actors involved, e.g. Member States under shared management arrangements, external evaluators, etc.

The Directorates-General's reporting on the achievements of the year

Agriculture and Rural Development

8.5. In its Annual Activity Report, DG AGRI presented in a table, for each ‘general objective’, figures on the current situation of the ‘impact indicators’ against the targets set. DG AGRI reported positive results in the field of sound financial management. The DG referred to the new obligation for Member States to assess their Land Parcel Identification Systems, and to the possibility for Member States to ask their certification bodies to confirm, after re-performance checks, the reliability of the control statistics. In such cases, the Commission would limit any financial corrections to the error rate derived from the control statistics concerned.

8.6. For Rural Development, DG AGRI reported progress in the implementation of the 94 programmes, with 33,9 billion euro of a total budget of 96,2 billion euro executed, although at a various pace between Member States and measures. As 2010 was the midpoint of the programming period, DG AGRI mentioned that a stocktaking exercise provided input to the reflections on the future of the rural development policy, including 90 mid-term evaluations of the Rural Development programmes submitted to the Commission by the end of the year.

8.7. The ‘specific objectives’, the related ‘results indicators’, and the main policy outputs delivered for each Activity-Based Budgeting activity (ABB) were included in Annex 6 to the Annual Activity Report.

Regional policy

8.8. In the Annual Activity Report, the Director-General reported that DG REGIO delivered on its operational priorities set for the year in its Management Plan and continued to make significant progress towards its long term and multiannual objectives. More generally, DG REGIO reported that Cohesion Policy (9) had demonstrated its effectiveness and consolidated evidence on its added value. 8.8. In its Annual Activity Report, the Directorate-General for Regional Policy (DG REGIO) reports performance on several aspects: budgetary execution, policy development, operational achievement linked to the quality of intervention and the implementation of the recovery package, sound financial management, etc. The operational priorities, defined in the Management Plan (MP), are one of the elements to assess performance.

8.9. DG REGIO provided further information on the controls in place to ensure economy, efficiency and effectiveness. Regarding the implementation of operational programmes, DG REGIO assessed that 73 % of the amounts decided for 2007-2013 had been implemented satisfactorily based on four criteria (progress in project selection procedures, major projects preparation and implementation, financial engineering instruments: setting-up and investment, payments to beneficiaries). In addition, DG REGIO presented recent evidence of the added value of the Cohesion Policy quoting the ex post evaluations of ERDF programmes 2000-2006, the conclusions of the ‘Experts evaluation network’ on the 2007-2013 programming period and the fifth Cohesion Report (10).

8.10. DG REGIO presented the latest developments regarding the implementation of the relevant programming periods separately for the European Regional Development Fund (ERDF), the Cohesion Fund, the pre-accession interventions and the European Union Solidarity Fund and included related performance indicators in Annex 7 to the Annual Activity Report.

Research and Innovation

8.11. In the Annual Activity Report, the Director-General recalled the objective of the Commission and the Member States to drive forward an integrated and world-class research system in Europe. The situation presented was that overall the EU scientific and technological competitiveness remained strong, accounting for 23,4 % of the world’s total research investment, 22 % of the researchers, 32,4 % of all the high impact publications and 31,3 % of all patents. However, there was a decline in favour of the rise of the emergent economies research investments.

8.12. Regarding the Seventh Framework Programme for Research and Technological Development (FP7), the EU’s main instrument to implement and support EU research policy, DG RTD reported to have concluded a total of 38 (11) calls for proposals in 2010 and to have committed over 5,2 billion euro.

8.13. The 2010 interim evaluation of FP7 concluded that it contributed to the development of European Research Area and its geographical outreach in cross-border cooperation and its promotion of research excellence continued to make a difference to the European research landscape.

8.14. The Annual Activity Report also presented the main achievements for each activity and reported on objectives and related indicators in its Annex 8.

The Court’s observations on Directorates-General's reporting

Relevance of performance information

A coherent set of objectives, indicators and targets focussing on effectiveness, but results need to be more thoroughly analysed

8.15. The DGs examined presented in the Management Plans a set of objectives, indicators and targets, focussing mainly on effectiveness. General objectives were defined at strategic policy level. Their achievement was measured using impact indicators and related targets. Specific objectives defined at operational level were linked to results indicators and targets. 8.15. The Commission considers that the Annual Activity Reports (AARs) examined by the Court are in compliance with the requirements of Article 60(7) of the Financial Regulation. The set of objectives, indicators and targets in the Management Plan has been designed to focus on policy or programme effectiveness.

8.16. In the Annual Activity Reports however, the differences between the targets and the achieved results were not analysed (DG REGIO, DG RTD) or only partially (DG AGRI). Thus, the information presented in the Annual Activity Reports was not sufficient to understand why a target was not achieved or significantly exceeded. 8.16. The standing instructions ask the Commission services to justify any discrepancy between results achieved and planned. For some cases, although a full assessment of the differences between the targets and the achieved results was not presented, data on the current situation is provided annually showing clearly the trend towards attainment of the objectives. The Commission agrees that departure of actual achievements from the planned targets could be better explained in future Annual Activity Reports, and, in the future, particular attention will be paid to further developing the analysis of the achievements against the targets set.

Currently the Management Plan does not foresee objectives and indicators to measure economy and efficiency

8.17. In the Management Plan, the DGs did not set objectives and related indicators to assess economy (costs of inputs) or efficiency (relation between inputs, outputs and results), either at the level of the EU policy, or at the level of expenditure under direct control of the DG. Thus, it was not possible in the Annual Activity Report to assess whether the achievements presented were the result of an economic and efficient management of resources or whether any progress had been recorded in this area. 8.17. Within the implementation of Activity Based Management, the MP was the tool chosen by the Commission for its services to translate the Commission's long-term strategy into general and specific objectives. Concerning the general objectives, the standing instructions invite the services to limit their number, to align them in particular with the EU 2020 Strategy and to define priorities amongst them. On the side of the specific objectives, services are instructed to set objectives that are aligned with the general objectives and that address the direct effect of the EU actions on the targeted population. In conclusion, the MP focuses on policy effectiveness (the degree to which objectives are achieved). On the side of the reporting, the Commission services report, in their AARs, on effectiveness, efficiency and economy aspects of their control systems, in line with Article 60(7) of the Financial Regulation. Part 2 of the AAR focuses on the effectiveness of the internal control system while Part 3 includes key indicators on legality and regularity which are established under the format ‘inputs, outputs, results and impact’.

In some areas targets not sufficiently quantified or specific

8.18. For DG AGRI four out of 10 targets for impact indicators, and for DG RTD three out of ten, were not quantified, mentioning only a trend. Explanations were not provided for the basis on which the targets were set for five impact indicators out of 10 in DG AGRI, and for four out of 10 impact indicators as well as all results indicators for cooperation in the different research areas in DG RTD. For DG REGIO, while the targets for impact and results indicators were quantified, most operational priorities and outputs for 2010 set in the Management Plan were not specific (12) or measurable. 8.18. The Commission standing instructions do not require all indicators to be quantitative nor to explain the targets as the MP is primarily a tool for management. Indicators need to be measurable to monitor progress towards achieving the set goals; and trends are adequate for this purpose. For DG REGIO, general and specific objectives linked to the activity-based budgeting (ABB) activities are ‘SMART’ and the definition of policy outputs is also clear. DG REGIO has introduced so-called ‘operational priorities’. These are aimed at introducing a management style based on a ‘systems thinking’ approach focusing on increasing quality. As such, the focus is more on processes and progress, rather than on achieving specific outputs and targets.

8.19. DG RTD used similar indicators in seven different Activity-Based Budgeting activities financed and did not consolidate their results. Overall assessment was therefore difficult. DG RTD’s Annual Activity Report did not clearly separate the impacts of projects financed through Framework Programme under its direct responsibility from those of its action to coordinate the research efforts of Member States under a common and coherent EU policy for research, where the Commission’s influence is indirect only. 8.19. Even though the same indicators are used for several activities, the standing instructions ask the Commission services to present their objectives and indicators by ABB activity, and not consolidate information across activities. It is extremely difficult to separate the impacts according to the nature of the Directorate-General for Research and Innovation (DG RTD) intervention, since impacts refer to ultimate changes in EU society. These societal changes depend essentially from the action of other actors (Member States, regions, undertakings …).

Interim milestones for multiannual targets not defined in some areas

8.20. In regional policy, research and to a lesser extent parts of agriculture, while DGs had set objectives and targets over multiannual programming periods, they did not use appropriate interim milestones. 8.20. For multiannual programmes it is not always meaningful to set interim milestones, as impacts will only be reached in a non-linear way at the end of the programming period. According to the Commission standing instructions, milestones are only needed for general objectives. The annual reporting on inputs, outputs and results, i.e. the monitoring of the progress in programme implementation, does allow for a regular assessment of the progress towards the attainment of the objectives. In addition, in the area of rural development, the mid-term evaluation of the rural development programmes (RDPs) provides a further in-depth independent quantitative and qualitative view of whether programmes are on track to achieve their objectives and what adjustments might be needed. For indicators measuring the ultimate goals of the research and innovation policy at EU level and for those of the Activity ‘ERA Development’, data are usually updated yearly but with a certain time lag of 2 or more years depending on the indicator. The Court's suggestion to provide more explicit explanations on whether achievement of objectives is considered on track could be explored in the future. See also reply to paragraph 8.21.

8.21. For multiannual programmes, the standard pace of progress might be relatively slow, so that an achievement of less than 10 % of the target after a number of years could in certain cases be considered normal, while in other cases signal delays. For example in the area of ERDF, the target for additional population covered by broadband access in six Member States was set at 10 million over 2007-2013. The latest known result was 550 000 for 2007-2009, representing 5,5 % of the overall target after 3 years out of 7. There was no indication on whether this level was considered sufficient to reach the multiannual target. In such instances, it will not be possible, till the very end of the period, to check whether the DGs are still on track towards the achievement of multiannual targets. 8.21. The pace of progress for multiannual programmes can indeed be relatively slow. The reasons are twofold: the implementation of the programmes entails initial capacity building and the required impact takes several years to gather speed. Regarding the European Regional Development Fund (ERDF) support to broadband access, the 2007-2013 targets mentioned for the various result indicators will not be achieved in 7 years but at the end of the eligibility period 2 years later. DG REGIO's has highlighted in key documents, all mentioned in the Annual Activity Report, the need for Member States to accelerate achievement of the EU 2020 objectives for broadband access (31). See also reply to point 8.20.

Description of policy achievements provided limited information on results and impacts

8.22. Along with performance indicators, Directors-General described in the Annual Activity Reports the year’s ‘policy achievements’. The text tended to focus more on outputs (what the DG did, describing actions taken, their purpose, the products delivered), and on inputs (gross amounts of resources involved), rather than on the assessment of the outcomes of the actions (results on direct addressees and/or impact of the programmes). 8.22. It is inherent to the annual nature of the annual activity report cycle that, within the multiannual period during which the policy and/or programme is being implemented, in the management part of the annual activity report the Authorising Officer by Delegation (AOD) focuses on the inputs and outputs which are under his management scope rather than already on the long term results and impacts of the policies and programmes, which can only be known at the end of the programme and which may also be influenced by other factors outside his management scope. See also reply to paragraph 8.23.

8.23. This was especially the case for DG AGRI, which provided, in its description of its policy achievements, almost no indication on the outcomes of its actions (13). In comparison, some more significant information on results and impacts was provided by DG REGIO (qualitative assessment of the implementation of the operational programmes), and, to a lesser extent, by DG RTD (presentation of key results of the FP7 interim evaluation). 8.23. Extensive evaluation assessments are more appropriately done via dedicated evaluation reports, which are made across DGs and involve more than one Authorising Officer by Delegation (AOD). The format of the AAR, because of its nature as a management report, is not suited to include extensive reporting on the results of evaluation activities. Nevertheless, if one of those evaluations would provide indications that policy and/or management aspects should be reconsidered, then action would be taken. Only in such cases, this would then also be signalled via the AAR (cf. exception-based reporting). In the description of the Directorate-General for Agriculture and Rural Development (DG AGRI) policy achievements the focus was set in reporting on the achievement of the priorities while adding some quantifiable data and also some relevant qualitative examples to illustrate such achievements per ABB activity. However, the information was presented taking into account the target audience — the general public — and therefore some concessions were made for easier readability, with the bulk of the performance information on results given in Annex 6 to the AAR.

Comparability of performance information

For DG AGRI and DG RTD, the objectives, indicators and targets set in the Management Plan were generally the ones used for reporting

8.24. Regarding DG AGRI and DG RTD, the objectives, indicators and targets announced in the Annual Management Plan were generally the ones used to report on performance in the Annual Activity Report. There was however a small number of instances (14) in which there was no explanation for the modification of indicators or targets, resulting from various reasons, like the choice of a more specific and measurable indicator, new policy priorities, or updated information. 8.24. DG AGRI considers that there is a good overall stability over the years as regards indicators and targets. As underlined by the Court, the change in indicators and targets is an exception resulting from various reasons such as the choice of a more specific and measurable indicator and/or updated information. In the future, it will be paid particular attention to make the rationale for changes in indicators/revision of targets more explicit. All instances of changes between objectives, indicators, and targets in the Management Plan and in the Annual Activity Report of DG RTD are justified. This justification could indeed be made more explicit in future AARs.

For DG REGIO, explanations were often not provided in the Annual Activity Report on the adjustments made to indicators and targets

8.25. The objectives announced in the 2010 Management Plan were the ones used to report on performance in the Annual Activity Report. However, when DG REGIO wanted to reflect policy evolution and adjusted the indicators and targets, the corresponding explanations were not presented in the following 14 out of 16 adjustments examined: 8.25. In general, there is a good stability over years as regards indicators and targets for DG REGIO. In the 2010 Annual Activity Report, efforts have been made to present evolution of the known results to progressively attain the multiannual targets. Revisions of indicators and targets reflect the ongoing progress towards reporting on and using up-to-date information available in official documents — and are identified by corresponding footnotes, with as much information as deemed appropriate. This process was explained in the 2009 AAR (page 21) stating that the core-indicators would be progressively used to follow-up the achievements of Cohesion Policy. As a result of the crisis, some programmes adapted their strategies and funding allocations across priorities which had consequences for the related targets, as explained in page 7 of the Annual Activity Report. DG REGIO will seek to ensure that such modifications are captured in the AAR.

— At the level of the seven impact indicators, one target had changed between the Management Plan and the Annual Activity Report without providing explanations for the difference.

— Regarding the ERDF results indicators, for instance, six (15) were added and one (16) was dropped without explaining the reasons for these adjustments. — Several environmental result indicators have been added for the ERDF activity (reduction of greenhouse gas emissions, flood protection, etc.). It is a positive revision in line with paragraph 8.18.

— Six ERDF targets were changed without explanation. They included the addition of a target of 170 000 SME support projects co-financed over the 2007-2013 period. For the five other changed targets for ERDF indicators (17), different Member States were covered than at the planning stage. For instance, the target for ‘additional population served by new/renovated wastewater projects’ was set at 31,5 million for 16 Member States in the Management Plan, and became 12,5 million for 13 Member States in the Annual Activity Report. The differences were not explained in the report. — The Management Plan figures refer to figures at the time of negotiations, while in the Annual Activity Report, DG REGIO was able to report on targets and actual achievements as reported in the 2009 Annual Implementation Reports. Where appropriate, footnotes have been added to give additional data and keep the perimeter mentioned in the 2010 MP). The new target of 170 000 SMEs represents an improvement of the reporting process as it was initially not possible to set a target when the Management Plan had been drafted. In the absence of complete indicators at EU level, it is acceptable to present indicators for some Member States only according to Commission instructions and this situation has been accepted by the budgetary authority in the past. In the Management Plan, the targets and the latest known results were sometimes presented for two different series of Member States. In order to improve the comparability of the latest known results and the targets DG REGIO has adjusted the figures using the same group of Member States.

Reliability of performance information

17 out of 31 indicators traced back to a reliable source of information

8.26. The Court examined a sample of 31 indicators selected over the three DGs (18). For 17 of these, the information on achieved results could be traced back to a reliable source. Generally it concerned data coming from external and recognised providers for statistics or internally managed databases, including accounting information. 8.26.-8.27. Footnote 19: The indicator mentioned by the Court indeed refers to the ‘Number of research infrastructures of world class relevance launched jointly at EU level’ and will be amended accordingly.

Weaknesses in the control system impairing reliability in the areas of regional policy and rural development

8.27. Out of the 14 other indicators, eight results indicators for the European Regional Development Fund (ERDF) and four impact indicators for rural development (19) were consolidated by the respective DGs on the basis of information which the Member States provided on the implementation of the programmes. The Commission's ability to ensure that information supplied was reliable and comparable was hampered by legal limitations (see paragraph 8.28) and practical constraints (see paragraph 8.29). 8.27. See reply to paragraph 8.29.

8.28. In the area of the ERDF, there was no legal requirement to implement a common set of performance indicators, so that managing authorities were not obliged to use the set of ‘Core indicators for ERDF and Cohesion Fund’ defined in the ‘Indicative guidelines on evaluation methods: monitoring and evaluation indicators’ by the Commission in August 2006. Consequently, there was a high risk that the data collected on the Core indicators and used in Annex 7 to the Annual Activity Report might not be exhaustive or comparable (20). 8.28. The constraints identified by the Court relate to matters that should be put in perspective. DG REGIO has continuously developed and improved arrangements for reporting on performance. It has gone beyond regulatory requirements intended by the legislator, in particular with the establishment of ‘core indicators’. However, DG REGIO is fully aware of those weaknesses and has been working to improve reporting on the recommended core indicators through working collaboratively with the Member States. Adjustments will be introduced in the future legal bases.

8.29. For both ERDF and rural development data, the Commission did not verify the reliability of the data provided by the bodies in charge in the Member States, beyond plausibility checks (21). The information received through the information technology systems used for reporting to the Commission by the Member States was aggregated at programme or measure level. The Commission had no direct access to data at the level of the projects so that it could not perform more systematic in-depth checks. 8.29. DG REGIO is checking the figures received from Member States (MS) and using its strong partnership with them and Regions to encourage and convince them to report more reliable figures. Nonetheless the legal requirements place the main responsibilities on Managing authorities in terms of project selection, objective setting and detailed monitoring. In terms of tracking the thematic use of EU financing the results of the 2009 annual programme reporting exercise has been discussed in different forums involving national authorities (COCOF, technical, annual meetings or monitoring committees) resulting in corrections in the data. The Commission's quality checking of the reporting in this information system will be continued and intensified. For rural development, the Commission established a Common Monitoring and Evaluation Framework (CMEF) for the 2007-2013 programming period. It provides for the first time a single framework for monitoring and evaluation of all rural development interventions on the basis of common indicators. The Commission devoted a lot of efforts, in cooperation with the Member States, to fine-tune the quality and reliability of the information provided under this framework. The mid-term evaluation of rural development programmes suggests that difficulties persist in quantifying the impact indicators for several programmes. Further work on these is ongoing, in close cooperation with Member States, both in relation to the current programming period, and in preparation for the future. Within this framework, the responsibility for the provision of reliable monitoring data relies on the Member States, in line with the repartition of competences under shared management. Considering the high number of projects and the broad range of Member States' project specificities, it is neither feasible nor cost effective for the Commission to verify at project level data provided by the Member States.

THE COURT’S SPECIAL REPORTS ON PERFORMANCE

Introduction

8.30. The Court’s Special Reports examine whether the EU’s intervention was managed in accordance with the principle of sound financial management. When choosing the topics for Special Reports, the Court considers the materiality of the audited area, the risks to sound financial management, previous coverage of related topics by the Court and other evaluations, and the relevance of the subject, i.e. the extent to which the audit results will help improve the effectiveness, efficiency and/or economy of EU expenditure.

8.31. The Special Reports adopted by the Court in 2010 are presented as following:

Special Reports adopted by the Court of Auditors in 2010 (22) — No 1/2010 ‘Are simplified customs procedures for imports effectively controlled?’ — No 2/2010 ‘The effectiveness of the Design Studies and Construction of New Infrastructures support schemes under the Sixth Framework Programme for Research’ — No 3/2010 ‘Impact Assessments in the EU institutions: do they support decision-making?’ — No 4/2010 ‘Is the design and management of the mobility scheme of the Leonardo da Vinci programme likely to lead to effective results?’ — No 5/2010 ‘Implementation of the Leader approach for rural development’ — No 6/2010 ‘Has the reform of the sugar market achieved its main objectives?’ — No 7/2010 ‘Audit of the clearance of accounts procedure’ — No 8/2010 ‘Improving transport performance on trans-European rail axes: Have EU rail infrastructure investments been effective?’ — No 9/2010 ‘Is EU Structural Measures spending on the supply of water for domestic consumption used to best effect?’ — No 10/2010 ‘Specific measures for agriculture in favour of the outermost regions of the Union and the smaller Aegean islands’ — No 11/2010 ‘The Commission's management of General Budget Support in ACP, Latin American and Asian Countries’ — No 12/2010 ‘EU Development Assistance for Basic Education in Sub-Saharan Africa and South Asia’ — No 13/2010 ‘Is the new European Neighbourhood and Partnership Instrument successfully launched and achieving results in the Southern Caucasus (Armenia, Azerbaijan and Georgia)’ — No 14/2010 ‘The Commission’s management of the system of veterinary checks for meat imports following the 2004 hygiene legislation reforms’

8.32. The 14 Special Reports adopted by the Court in 2010 made the following observations concerning the performance achieved and the process of producing results by spending EU funds, from the initial planning to reporting on performance.

Court’s observations concerning the process of producing results

Strategic planning

8.33. In a number of its Special Reports (SRs) (23), the Court highlighted the principles which govern good planning: initiatives should be set in a strategic context and prioritised; specific, measurable, achievable, relevant and timed (SMART) objectives should be established, and consistently articulated in the various planning documents; good ex-ante impact assessment can play an important role in the decision-making process.

8.34. In this respect, the Court found in the area of development assistance for basic education inappropriate strategic planning in capacity development initiatives (SR No 12/2010, paragraph 69), insufficient prioritisation and articulation of objectives in the neighbourhood policy (SR No 13/2010, paragraph 68), and, for the Leader approach, a lack of specific, measurable objectives, achievable within a set time frame (SR No 5/2010, paragraph 26). In its report on the Leonardo da Vinci mobility scheme, the Court found that the Commission had not finalised the setting up of SMART objectives and impact indicators to evaluate the success of the programme against its objectives and as a result the Commission had not been in a position to measure how the objectives were being met 3 years into the programme (SR No 4/2010, paragraphs 48, 50, 61 and 62). 8.34. Strategic planning in capacity development is reliant on a clear demand from the partner country and government leadership. Lessons learnt in this area show that, without this ownership, external institutional support is unlikely to be used as the foundation for effective capacity development. With regard to the Leonardo da Vinci mobility scheme, the Lifelong Learning Programme Committee adopted the indicators in its meeting of 17 June 2010. The Commission will provide a first report on these indicators to the Committee in the course of 2011. As regards the Leader approach, the Commission agrees that the objectives of the individual local development strategies should be established in line with SMART criteria to the extent possible. In order to help local action groups in this task, rural development funding supports through the sub-measure ‘acquiring skills’, both assistance and training activities.

8.35. The Court reported, however, improvements in the definition of objectives in the area of budget support to developing countries (SR No 11/2010, paragraphs 56 and 96), and an instance of good practice in the area of ‘better regulation’ where impact assessments had effectively supported decision-making in the EU (SR No 3/2010, paragraph 87). 8.35. The Commission welcomes the recognition of the improvements made in the definition of budget support objectives. The EDFs' Annual Report also confirms the significant progress made by the Commission in demonstrating compliance with the eligibility criteria set by the Cotonou Agreement and in introducing formal assessment frameworks for public finance management performance.

Identification of needs

8.36. In some of its observations (24), the Court also illustrated that, if good results are to be produced, it is important clearly to identify the needs which the programmes are intended to fulfil.

8.37. Thus the Court observed that the definition of Priority Projects for the trans-European rail axes should reflect present and anticipated needs (SR No 8/2010, paragraph 63). 8.37. With regard to the definition of Priority Projects for the trans-European rail axes, experience has shown that arriving at a definitive description of the main trans-European rail axes, while correct as an aspiration, is particularly difficult as they are in a constant state of flux depending on migration, trade patterns and the geopolitical context. Studies to analyse existing and expected traffic flows have been carried out, both for individual projects and for the network and have yet to lead to conclusive results. As a result, such a definition is not currently possible and Priority Projects should continue to be based on political agreements between the Council and the European Parliament based on the best available evidence.

8.38. It also pointed out that a good analysis of needs avoids fixing objectives which go beyond what is really required, while, at the same time, encourages alternative, less costly solutions (SR No 9/2010, paragraph 72).

Link between objectives and measures

8.39. In a number of its reports (25) the Court highlighted that the link must be clear between the objectives pursued and the measures chosen to reach those objectives.

8.40. When this link is not specified, it is unlikely that the design of the measures selected will be capable of achieving the objectives pursued (see SR No 10/2010, paragraph 34, concerning the specific case of the smaller Aegean islands). In the area of ‘better regulation’, the Court observed that the impact assessment reports did not provide a standardised presentation of how the objectives and expected outcomes of the proposed intervention could be achieved with the intended delivery mechanisms, and with regard to expenditure programmes, the estimated budget (SR No 3/2010, paragraph 60). 8.40. The Commission agrees that the support programme for the smaller Aegean islands could be improved and is working in cooperation with the Greek authorities to this end.

Relevant, sufficient and proportionate means

8.41. Some observations of the Court (26) underline the importance of an adequate selection of means to obtain set objectives.

8.42. In the area of development assistance for basic education in sub-Saharan Africa the Court pointed to the difficulty that the Commission has experienced in assigning staff with the appropriate profiles to the delegations, which weakens their ability to manage the programmes effectively (SR No 12/2010, paragraph 66). In the different context of the clearance of accounts procedure, the Court signalled that the corrections imposed, to be really effective, should have directly preventative or dissuasive consequences on those responsible for the irregularity (SR No 7/2010, paragraph 90). 8.42. The Commission acknowledges that adequate qualified expertise in the field of education is essential. When education is a priority for a delegation, the Commission takes steps to ensure that adequate expertise is available, for example by delegating under the EU division of labour policy responsibility to a Member State which has relevant sectoral expertise, or by providing appropriate expertise from Headquarters through the geographical directorates with the support of thematic units. It is also planned that thematic networks will be developed to better link expertise in the thematic and geographical directorates in Headquarters with Delegations. The conformity clearance is designed to exclude from EU financing expenditure which does not comply with EU rules. It is not a mechanism by which irregular payments to beneficiaries are recovered, which according to the principle of shared management is the sole responsibility of Member States. Where undue payments to beneficiaries can be identified as a result of the conformity clearance, Member States are required to follow them up with recovery actions against these beneficiaries. Where recoveries are not needed because the financial correction relates to deficiencies in a Member State's management and control systems, the corrections are an important means to improve those systems and thus to prevent or detect and recover irregular payments to beneficiaries. Moreover, agricultural legislation provides for effective, dissuasive and proportional sanctions to be imposed on beneficiaries who have received irregular payments.

Budget and time frame

8.43. The Court pointed out in its reports (27) the importance of planning during the implementation phase of projects, to avoid overspending or delays.

8.44. In this respect, the Court observed that, regarding EU rail infrastructure investment, projects with less thorough and detailed preparation faced a higher risk of experiencing more significant cost escalations (SR No 8/2010, paragraph 46). The Court also noted that delays in implementing projects concerning domestic water supplies caused by additional requests imposed by environmental impact assessments, difficulties in obtaining administrative permits, inaccurate estimates or calculations could have been partly avoided by better planning (SR No 9/2010, paragraph 51). 8.44. The Commission has repeatedly emphasised the importance of project planning by national and regional authorities in the context of Cohesion Fund projects and major projects under the Structural Funds. In the context of the 2007-2013 structural funds programming period the technical assistance facility JASPERS has been established specifically to support, mainly in the EU 12, better project design and planning. However, planning is just one element affecting overspending and delays. As the Court also concluded in paragraph 45 of its Special Report on EU rail infrastructure investment that, ‘In nearly all cases, these [cost] escalations arose for reasons linked to unforeseeable factors that came to light during the construction phase, such as unexpectedly difficult geographical conditions, environmental protection requirements, safety requirements and higher than expected bids from contractors’.

Controls and performance monitoring

8.45. Some of the Court’s observations (28) encouraged managers to improve systems of internal control, and to monitor performance by setting up appropriate mechanisms to record information and data on the achieved outcomes or on the level of performance.

8.46. For example, the Court observed that the reviews carried out to monitor EU-funded interventions in basic education in sub-Saharan Africa focussed in general on the processes and financial issues linked to specific disbursement conditions rather than on education outcomes (SR No 12/2010, paragraph 62). Similarly, it observed that the Local Action Groups responsible for implementing the Leader programme did not collect data on or assess whether local strategy objectives were met (SR No 5/2010, paragraph 31). 8.46. In relation to education sector policy support programmes financed by sectoral budget support, progress against outcome indicators (enrolment, completion, survival, and learning achievements in some cases) is assessed in Joint Reviews with other donors, led by the partner country. When agreeing performance monitoring arrangements with partners, the Commission considers that a good mix of outcome and process indicators provides the necessary comprehensive picture of the progress of the country in implementing its sector policy. In relation to Leader, the Commission has provided further guidance to Member States on how to better monitor, assess and improve the Local Action Groups' (LAGs) strategies and their respective achievements. It has also launched in early 2011 a Focus Group on the quality of local development strategies, including monitoring and evaluation issues, under the European Network for Rural Development. The results, which will be available by the end of the year, will provide further tools and good practice to both Member States and LAGs.

8.47. The Court also noted, however, efficient aspects of the supervisory and control systems, such as the role of the verifications carried out by the Food and Veterinary Office for maintaining a necessary control pressure (SR No 14/2010, paragraph 67), or the development by the Commission of a sound approach for controls on simplified customs procedures (SR No 1/2010, paragraph 81). 8.47. The Food and Veterinary Office is constantly assessing both the effectiveness and the efficiency of its control work to ensure that resources are put to the best use.

8.48. The Court underlined in one of its reports (SR No 10/2010, paragraph 40) the usefulness of an annual assessment of the programmes’ implementation, indicating that such assessment enabled managers to bring effective modifications to ongoing programmes. 8.48. The Commission has set common performance indicators whose assessment will help further improving the POSEI and the smaller Aegean Islands schemes.

Court’s observations concerning achievements and reporting

Achievement of the objectives set

8.49. Two of the Court’s reports conclude positively on results achieved through interventions financed by the EU budget (SRs No 8/2010 and No 10/2010). The Court found that EU co-financed rail infrastructure projects delivered the planned infrastructure to specification, and, once completed, have created new and improved rail transport possibilities on key sections of the Priority Projects (SR No 8/2010, paragraph 65). Regarding specific measures for agriculture in favour of the outermost regions of the Union and the smaller Aegean islands, the programmes were in general implemented effectively and thus met the needs of these regions (SR No 10/2010, paragraph 82).

8.50. Other special reports present more mixed results, with some important objectives not met (SRs No 6/2010, No 9/2010, No 12/2010 and No 13/2010). The Court observed that where there are conflicting objectives it is difficult for them to be achieved simultaneously (SR No 6/2010, paragraphs 93 and 99). Whilst structural measures spending has contributed to improving the supply of water for domestic use, better results could have been achieved at a lower cost to the EU budget (SR No 9/2010, paragraph 71). 8.50. The objectives of the 2006 reform of the EU sugar market are directly linked to the objectives of the common agricultural policy (CAP), which are enshrined in the Treaties. It is not a shortcoming of the reform to have partly diverging objectives, but rather a result of its comprehensive scope. The Commission considers that the sugar reform has successfully managed the restructuring of the sector, providing it with a long-term policy framework and considerably improving its competitiveness.

Reporting on performance

8.51. In its reports, the Court drew attention (29) to the need for appropriate reporting structures and relevant, comparable and reliable data to enable assessment of the correspondence between the objectives fixed and the results obtained.

8.52. In its report on the Leonardo da Vinci mobility scheme, the Court noted that as the annual work programme was structured differently from the annual activity report, it was not possible to make a meaningful comparison of results against planned performance (SR No 4/2010, paragraph 41). The Court also observed that the Commission and the Member States were not able to establish the effectiveness of Leader programmes in delivering of rural development outcomes in the absence or unreliability of data either in absolute terms or in relation to the effectiveness of other delivery methods (SR No 5/2010, paragraph 100). 8.52. With regard to the Leonardo da Vinci mobility scheme — as announced in the action plan in reply to the Court's report — an internal working group of DG Education and Culture revised the requirements and form of the Yearly National Agency Report in the second quarter of 2011. The revised report form is now in line with the structure and content of the National Agency work programme revised for 2011. As a consequence the Yearly NA Report on the year 2011 (to be received by the Commission in 2012) will allow meaningful comparison of results against planned performance. The mid-term evaluation of Leader+ took place at an early stage in LAG strategy implementation. However, the ex-post evaluation of Leader+, which was finalised in December 2010, provides useful information on the effectiveness and efficiency of Leader interventions. This information will be useful in the context of the ongoing work on the post-2013 CAP policy framework.

CONCLUSIONS AND RECOMMENDATIONS

8.53. As noted at the beginning of this chapter, the Financial Regulation provides that the Annual Activity Reports shall indicate the results of the operations by reference to the objectives set, the risks associated with these operations, the use made of resources provided and the efficiency and effectiveness of the internal control system (30). The Court observed that the differences between planned targets and achievements were often not analysed (see paragraph 8.16) and that the framework for reporting on effectiveness did not cover economy and efficiency of the spending (see paragraph 8.17). 8.53. The Commission agrees that departure of actual achievements from the planned targets could be better explained in future Annual Activity Reports. According to Article 60(7) of the Financial Regulation, it is the efficiency and effectiveness of the internal control system which has to be reported in the AAR and not the efficiency and effectiveness of the EU actions. The content of the AAR is defined according to the requirements of this provision. In any case, thanks to the line of consistency provided by the ABB nomenclature, the human and financial resources (as shown in Annexes 2 and 3) allocated to each ABB activity are clearly identified and can be linked with the indicators in the corresponding Annex, thus making it possible to assess the economy and efficiency in the management of the resources available.

8.54. The absence of appropriate interim milestones hindered the assessment of whether the progress made could be considered on track towards the achievement of multiannual objectives (see paragraphs 8.20 and 8.21). 8.54. For multiannual programmes it is not always meaningful to set interim milestones, as impacts will only be reached in a non-linear way at the end of the programming period. The annual reporting on inputs, outputs and results, i.e. the monitoring of the progress in programme implementation, does allow for a regular assessment of the progress towards the attainment of the objectives.

8.55. The Court noted that the objectives, indicators and targets set in the Management Plans were generally those used for reporting in the Annual Activity Reports (see paragraphs 8.24 and 8.25).

8.56. Legal limitations and practical constraints impaired the reliability of the planning and reporting information collected by the DGs from Member States regarding programmes under shared management (see paragraphs 8.27 to 8.29). 8.56. For rural development, the Commission established a Common Monitoring and Evaluation Framework (CMEF) for the 2007-2013 programming period. It provides for the first time a single framework for monitoring and evaluation of all rural development interventions on the basis of common indicators. Within this framework, the responsibility for the provision of reliable monitoring data relies on the Member States, in line with the repartition of competences under shared management. Considering the high number of projects and the broad range of Member States' project specificities, it is neither feasible nor cost effective for the Commission to verify at project level data provided by the Member States. See also reply to point 8.29.

8.57. The Court concludes that the quality of the planning phase is important in determining how the intended results can be obtained. Insufficient or weak planning, including the absence of SMART objectives, may produce delays, additional costs, and affect the results achieved (see paragraphs 8.33 to 8.44). 8.57. The Commission agrees on the importance of the planning phase and the need to rely on SMART objectives.

8.58. It is important for the management’s accountability that the results reported correspond to the objectives and indicators set in the management plan and that appropriate monitoring and control systems are in place to obtain reliable information for reporting on results (see paragraphs 8.45 to 8.48 and 8.51 to 8.52). 8.58. The Commission agrees that it is important for the management’s accountability that the results reported correspond to the objectives and indicators set in the management plan and that appropriate monitoring and control systems should be in place to obtain reliable information for reporting on results. The Commission is committed to constantly improving its monitoring and control systems to ensure that they are appropriate and adequate to deliver the necessary information reliably.

8.59. The Court makes the following recommendations: 8.59.

— Recommendation 1: Increased focus should be put on performance in the DG Annual Activity Reports, in particular by analysing differences between planned targets and achievements as well as by reporting on the economy and efficiency of EU funding (see paragraph 8.53). — Differences between planned targets and actual achievement against objectives set could indeed be better explained. However, MP and AAR are not intended to provide detailed information on economy and efficiency (See replies to paragraphs 8.17 and 8.53). For the programmes in the area of structural funds, the assessment of the economy and efficiency of the funding schemes is highly dependent on the willingness and the ability of Member States to report on issues like the unit costs or the links between outputs and categories of expenditure. This is not possible across all the programmes in a synthetic way under the current legal framework.

— Recommendation 2: The Commission services should define appropriate interim milestones for multiannual targets, so that progress can be assessed adequately (see paragraph 8.54). — For output, result and, where appropriate, impact indicators, interim milestones could indeed be defined. For outputs, it is possible to define milestones for multiannual targets at least at programme level. However, this seems more artificial for results indicators due to the different contributing factors. Nevertheless, the Commission is willing to include some further requirements in the future regulations asking for progress reports of the newly proposed partnership contracts in particular in respect of milestones set out in the performance framework.

— Recommendation 3: The Commission and the Member States within the context of their respective responsibilities under shared management should agree on consistent performance indicators and ensure the reliability of information on planned targets and achieved results (see paragraph 8.56). — Under shared management, the Commission plays a supervisory role by satisfying itself that the arrangements governing management and control systems are compliant, and by verifying the effective functioning of the systems. In this context, the Commission has already engaged such a process with Member States. For rural development, the Commission has already established a Common Monitoring and Evaluation Framework (CMEF) for the 2007-2013 programming period which provides for the first time a single framework for monitoring and evaluation of all rural development interventions on the basis of common indicators. The Commission intends to work more closely with the Member States to further improve the CMEF for the next programming period. A first seminar on CAP monitoring and evaluation is already scheduled for September 2011. For the Structural Funds, the Commission will propose to agree on common output indicators for the future regulation post 2013 and will agree a methodology for the identification of appropriate result indicators which must be context specific. The verification of planned targets and the reporting of achieved results will still be under the responsibility of Member States (see reply to recommendation 6).

— Recommendation 4: During the planning of EU expenditure programmes, the Commission and the Member States should pay greater attention to defining SMART objectives, as well as to identifying and mitigating the risks which may occur during implementation (see paragraph 8.57). — The Commission always endeavours to define SMART objectives and to pay attention to identifying and mitigating the risks which may occur during implementation. These efforts would be further re-enforced in the future.

— Recommendation 5: Accountability of the management should also be understood to include reporting on results with a correspondence between achievements expected in the management plan and achievements reported in the Annual Activity Report (see paragraph 8.58). — The Commission agrees that it is important for the management’s accountability that the results reported correspond to the objectives and indicators set in the management plan, as is already the case for the large majority of them, and that appropriate follow-up and control systems should be in place to obtain reliable information for reporting on results. The Commission should maintain a flexible approach to be able to disclose in the Annual Activity Report the latest data even if they were not anticipated in the Management Plan. However, it will present data so as to allow the comparability of the two exercises.